Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency

Large language models (LLMs) commonly employ key–value (KV) caching to avoid recomputing previous attention states during autoregressive inference. However, its measured performance benefit depends on workload characteristics and the memory environment. This study presents a controlled systems characterization across input context length, model-specific grouped-query attention configurations, and GPU-memory residency. Experiments use Qwen2.5-3B and SmolLM3-3B under verified fully resident conditions with fixed 128-token generation, prompt lengths from 128 to 2048 tokens, randomized cache-enabled/cache-disabled execution, GPU telemetry, and repeated paired validation. End-to-end generation speedup increases from 1.34× to 8.31× for Qwen2.5-3B and from 1.28× to 8.62× for SmolLM3-3B. At 2048 prompt tokens, 30 randomized paired trials confirm mean speedups of 8.10× and 8.51×, corresponding to runtime reductions of 87.65% and 88.25%, while physical GPU-memory utilization remains below 61%. SmolLM3-3B requires twice the theoretical KV-cache storage per token, yet both models show comparable context-dependent scaling in the fully resident regime. Because the models differ in characteristics beyond KV-head count, this cross-model comparison is descriptive and does not attribute the observed performance difference causally to GQA configuration. A separate Mistral-7B case study shows that apparent end-to-end speedup changes substantially under memory pressure; the underlying memory-system mechanism is not isolated. These results establish cache-related end-to-end generation speedup as workload- and system-dependent and demonstrate that meaningful benchmarking requires reporting context length, cache configuration, model and cache footprints, physical accelerator-memory residency, runtime, throughput, and relevant GPU-memory conditions during each inference experiment.