Analysis and Evaluation of Using Microsecond-Latency Memory for In-Memory Indices and Caches in SSD-Based Key-Value Stores

When key-value (KV) stores use SSDs for storing a large number of items, oftentimes they also require large in-memory data structures including indices and caches to be traversed to reduce IOs. This paper considers offloading most of such data structures from the costly host DRAM to secondary memory whose latency is in the microsecond range, an order of magnitude longer than those of currently available DIMM-mounted or CXL memory devices. While emerging microsecond-latency memory is likely to cost much less than DRAM, it can significantly slow down SSD-based KV stores if naively employed. This paper analyzes and evaluates the impact of microsecond-level memory latency on the KV operation throughput. Our analysis finds that a well-known latency-hiding technique of software prefetching for long-latency memory from user-level threads is effective. The novelty of our analysis lies in modeling how the interplay between prefetching and IO affects performance, from which we derive an equation that well explains the throughput degradation due to long memory latency. The model tells us that the presence of IO significantly enhances the tolerance to memory latency, leading to a finding that SSD-based KV stores can be made latency-tolerant without devising new techniques for microsecond-latency memory. To confirm this, we design a microbenchmark as well as modify existing SSD-based KV stores so that they issue prefetches from user-level threads, and run them while placing most of in-memory data structures on FPGA-based memory with adjustable microsecond latency. The results demonstrate that their KV operation throughputs can be well explained by our model, and the modified KV stores achieve near-DRAM throughputs for up to a memory latency of 5 microseconds. This suggests the possibility that SSD-based KV stores can use microsecond-latency memory as a cost-effective alternative to the host DRAM.