When Does Keeping Agent Sessions in GPU Memory Pay Off? A Rust Serving Engine and a Replay Study on Real Agent Traces

AI agents call a language model, run a tool, and call the model again with the whole historyattached. Keeping an agent’s KV cache in GPU memory while it waits on a tool avoids recomputingthat history, but holds memory other requests could use. Prior systems (InferCept,Pensieve, Continuum, TokenCake and others) model this trade-off and evaluate it against earlierversions of vLLM. We ask how it plays out against the current prefix caches of vLLM 0.33.0 andSGLang 0.5.23. We built a serving engine in Rust (about 1,700 lines) with per-session KV retention,cross-session prefix sharing and session-level admission, and replayed 24 real coding-agentsessions (791 turns) against all three engines with the same GPU, model and KV cache size, tworuns per configuration. With a fixed 0.5-second tool wait, close to the median gap measuredfor coding agents, our engine finishes the workload 1.31x faster than the best baseline (1.45xfaster than vLLM) and returns the first token about 5x sooner. With a 5-second wait the gain is1.13x (1.24x over vLLM). With a 60-second wait, all engines finish within 2% of each other andour engine is slower to the first token, consistent with prior reports that retention stops payingoff when agents are away long. Three pinning policies we tried, each with a pass/fail criterionwritten down before seeing its results, did not beat the baselines at 60 seconds.