Answers are grounded in this paper and cite their sources.
Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when LLMs are used in the common setting of multi-turn conversations, a growing log of the conversation history must be processed alongside any request by the serving system at each turn, resulting in repeated processing.
No takes yet. Share an insight, caveat, or question.
Yu et al. (2025) studied this question.