Perplexity Holds, Programs Break: Executable Correctness as a Blind Spot of KV-Cache Compression

KV-cache compression methods are evaluated almost exclusively with metrics that score token or string overlap, retrieval hits, or an extracted final answer. None of the widely used long-context suites check whether generated output actually runs, or whether a generated tool call is well-formed. We argue this is a systematic blind spot: lexical and retrieval metrics can stay flat while a compressed cache quietly breaks structured generation, because a single dropped or garbled token fails a unit test or a schema check but barely moves a substring score. We introduce kv-exec-bench, a small open benchmark that measures the two missing quantities, code unit-test pass@1 and tool-call JSON-Schema validity, under KV compression. It is built on top of NVIDIA's kvpress as a dependency, so any press works without modification. In a deliberately small, CPU-only study on a 0.5B-parameter model, we observe the predicted divergence on tool calling: for the best-behaved presses, the rate of parseable tool calls stays at 100 percent across compression ratios while the rate of schema-valid calls falls by half or more. A parse-only or string-only metric would report this regime as nearly lossless. We release the benchmark and all code under Apache-2.0, and frame this as a measurement-gap report plus a reusable benchmark rather than a large-scale empirical study.