Speeding Up Write-Read Orders in Disaggregated Storage

Modern data centers adopt disaggregated storage architectures via network storage protocols, such as NVMe-over-Fabrics (NVMe-oF), to achieve fine-grained resource elasticity and enhanced resource utilization. A client-side writeback cache is used to absorb write operations and buffering frequently accessed data, thereby reducing remote storage accesses and improving overall performance. However, when a cache miss occurs on the full cache, the system must perform an evict-and-fetch operation: evicting dirty entries to reclaim space before fetching newly requested data blocks. Existing systems perform the evict-and-fetch operation by sequentially executing write and read I/O operations, which reduces the concurrency and makes it challenging to fully utilize the fast network and storage devices. To overcome this problem, we propose Volley , a network storage protocol that enables the writeback cache to issue eviction and fetch operations simultaneously while guaranteeing the execution order of the write and read I/O operations. We implement the Volley protocol by extending NVMe-oF atop commodity network and storage hardware, and further employ the ACK free-rides and notification acceleration techniques to reduce the network overhead. We integrate Volley into two writeback caching systems ( V-Cache and V-TriCache ) and evaluate them using real-world workloads spanning KV cache offloading, cloud storage, and out-of-core computing scenarios. Evaluations show that V-Cache reduces P99 time-to-first-token (TTFT) by up to 38.4% in KV cache offloading scenarios and improves throughput by up to 6.84 × in cloud storage scenarios. V-TriCache reduces the total running time of Facebook Mixgraph by up to 16.7% compared with the state-of-the-art out-of-core computing system, TriCache.