Answers are grounded in this paper and cite their sources.
In LLM serving with prefix caching, several request orders can incur the same immediate prefill work yet leave different cache states. We give an exact method to score a known, finite continuation from each state under least-recently-used eviction and entry-time charging of whole prefix paths. The continuation can select its next request using current hits. Equivalent starting states can share scores, and a sufficient persistence test can determine a score without replay. In one previously examined Mooncake tool/agent condition (128 blocks, decision output), the two shortcuts reduced full replays from 267,672 to 46,610. The complete-state and focused-state implementations used the same shortcuts and agreed on replay counts in each of eight conditions, as expected. The complete-state implementation used more mean native CPU in six of the eight. In the highlighted condition the focused-state implementation took 66.59 microseconds per decision versus 89.00 for the complete-state implementation. Local caller wall time fell 16.9%. The policy saved 0.204% of modeled prefill work at 128 blocks and 1.001% at 32 on Mooncake, and zero on two external episodes. We have not established an end-to-end benefit for an independent planner.
No takes yet. Share an insight, caveat, or question.
Mohammad Ali Seyed Mahmoud (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: