When the Wrong Question Scores Higher: Lexical Overlap and the Limits of a Single Similarity Threshold in LLM Semantic Caching

This is the authors' abstract. We don't add key points for this paper.

Semantic caches reduce the cost and latency of large-language-model (LLM) applications by returning a stored response when an incoming query’s embedding is sufficiently similar to that of a previously answered query. In deployed products the decision is governed by a single similarity threshold, typically shipped as a default between 0.80 and 0.95. We ask how well such a threshold separates meaning-preserving rewordings, which should be served from cache, from minimally edited queries whose correct answer differs, which should not. We construct a labelled set of 243 English customer-support question pairs across eight domains: 101 heavily reworded paraphrases, 102 near-miss pairs that differ by a word or by word order but require different answers, and 40 lightly reworded paraphrases reported separately. Scoring every pair with a widely used open embedding model (Qwen3-Embedding-0.6B) by cosine similarity, we find the two core distributions inverted rather than merely overlapping: near-miss pairs score higher (median 0.801) than genuine paraphrases (median 0.691), giving an area under the ROC curve of 0.144 (95% bootstrap CI 0.097–0.195). At 0.85, the default threshold of one commercial service, the model matches none of the 101 paraphrases and 29 of the 102 near-misses (28.4%; Wilson 95% CI 20.6–37.8%). False matches reach zero only above 0.985, where 2 of 40 near-verbatim repeats remain. Similarity correlates with word-set Jaccard overlap (Pearson r = 0.695), including within each label. A query-instruction prefix halves false matches at 0.85 (12.7%) without changing the ordering. We argue that a threshold is a property of one embedding model on one traffic distribution, not a transferable constant, and we describe a labelled near-miss protocol for setting it. The pairs, scores and an analysis script that recomputes every statistic are released under CC BY 4.0. Preprint, not peer reviewed. Dataset: SemCache Near-Miss Pairs v1.0 (see Related works).