What question did this study set out to answer?

This research aims to improve cache serving efficiency in large language models (LLMs) by making eviction reversible and eliminating recompute needs.

May 26, 2026Open Access

EVOKE: A KV-Cache Memory Hierarchy with Recompute-Free Block Recovery for LLM Serving

Key Points

This research aims to improve cache serving efficiency in large language models (LLMs) by making eviction reversible and eliminating recompute needs.
Developed EVOKE to enable reversible cache eviction and optimize tensor recovery using similarity scores.
Implemented KV-paging primitives in a llama.cpp fork and tested on four model families using a single 16 GB GPU.
Evaluated recovery performance across benchmarks with comparisons to baseline methods.
Recovery-bearing policies achieved at least a 5.9 to 7.5× speed-up for save+load round-trips compared to re-prefilling on Qwen 2.5 7B, and 2.6 to 2.8× on Llama 3.1 8B.
Among benchmarks, recovery methods consistently achieved pass rates above 48%, outperforming recovery-less baselines which stayed below 43%.
At specific block sizes, EVOKE showed competitive performance against InfLLM, particularly leading at b=2048 with overlapping confidence intervals.

Abstract

Production LLM serving treats cache overflow as a dead end: under budget pressure, truncate the oldest history or re-prefill on every call. EVOKE (EVict and recOver KV cache Entries) makes eviction reversible. A recompute-free splice writes the saved K and V tensors back into the active cache with a single RoPE rotation; the cost is the tensor transfer. Similarity scores select which evicted block to recover; the splice then returns that block’s bytes at a new logical position, byte-identical to what the model originally attended. Recovery is keyed by block identity (where RAG would substitute re-encoded text). We add three KV-paging primitives to a llama.cpp fork (block save, block load with RoPE re-anchoring, per-layer attention capture), wrapped by a Python policy and evaluated across four model families on a single 16 GB consumer GPU. The save+load round-trip runs 5.9 to 7.5×faster than re-prefilling the same token range on the same engine (Qwen 2.5 7B), 2.6 to 2.8×on Llama 3.1 8B; load-only is 20 to 32×and 7 to 15×respectively. Across three benchmarks (NIAH, multi-fact, agentic), every recovery-less baseline stays at or below 43% multi-fact pass rate while every recovery-bearing policy reaches at least 48%, holding at b=1024 on hybrid Mamba/Attention Qwen 3.5 9B (68%) and MoE Qwen 3.6 35B-A3B (52%). Recency, StreamingLLM, H2O (Zhang et al., 2023), and SnapKV (Li et al., 2024) are same-harness reimplementations of their published configurations. Within the recovery-bearing cluster, a 15-seed sweep against InfLLM (Xiao et al., 2024b) (K=8, same splice) shows a budget-axis crossover: InfLLM wins at b=512 on both architectures (non-overlapping Wilson CIs), the two tie at b=1024, and EVOKE’s point estimates lead at b=2048 (CIs overlap, Section 7.3). The splice is the load-bearing contribution; scheduler choice and its Kare regime-dependent parameters.

Read Full Paperexternally

AI에게 질문

Bookmark

View Full Paper