SoftCache: A Software Cache for PCIe-Attached Hardware Accelerators

Hardware accelerators are used to speed up computationally expensive applications in many scientific fields. However, offloading tasks to accelerator cards requires data to be transferred between the memory of the host and the external memory of the accelerator card; this data movement frequently becomes the bottleneck for increasing accelerator performance. In this work, we explore the use of a software cache to optimize communication and alleviate the data-movement bottleneck by transparently exploiting locality and data reuse. We present a generic, application-agnostic framework, dubbed SoftCache, that can be used with both GPU and FPGA accelerator cards. SoftCache exploits locality to optimize data movement in a non-intrusive manner (i.e., no changes to the algorithm are necessary) and allows the programmer to tune the cache size, cache organization, and replacement policy toward the application needs. Each cache line can store data of any size, thereby eliminating the need for separate caches for different data types. We used a phylogenetic application to showcase SoftCache. Phylogenetics study the evolutionary history and relationships among different species or groups of organisms. The phylogenetic application implements a tree-search algorithm to create and evaluate phylogenetic trees, while hardware accelerators are used to reduce the computation time of probability vectors at every tree node. Using SoftCache, we observed that the total number of bytes transferred during a complete run of the application was reduced by as much as 89%, resulting in up to 1.7x (81% of the theoretical peak) and 3.5x (75% of the theoretical peak) higher accelerator performance (as seen by the application) for a GPU and an FPGA accelerator, respectively.