What type of study is this?

This is a Quantitative Study study.

October 8, 2025Open Access

Towards More Economical Context-Augmented LLM Generation by Reusing Stored KV Cache

Puntos clave

KV cache reuse significantly cuts down delay and cloud costs across various workloads with long context.
The proposed analytical model evaluates compute, storage, and network costs tied to KV cache usage.
Results indicate that effective KV cache management can enhance performance without increasing overall expenses.
Further development is needed to establish more economical context-augmented LLMs using KV caches.

Resumen

Across large language model (LLM) applications, we observe an emerging trend for reusing KV caches to save the prefill delays of processing repeated input texts in different LLM inputs. This has led to a broad design space, including colocating stored KV caches with (or close to) GPUs to various KV cache compression. However, a key question remains unanswered: can these delay reductions also be economically favorable? Specifically, we ask whether a developer can use public cloud services to store precomputed KV caches and reuse them to save delay without incurring more costs in terms of compute, storage, and network. To answer this question, we propose an validated analytical model for the cloud cost (in compute, storage, and network) of storing and reusing KV caches based on various workload parameters, such as reuse frequency, generated text lengths, model sizes, etc. Preliminary results show that KV cache reusing is able to save both delay and cloud cost across a range of workloads with long context. And we call more efforts on building more economical context augmented LLM by KV cache reusing.

Leer artículo completoexternamente

Preguntar a la IA

Me gusta

Guardar

Ver artículo completo