Razor: Learning What to Forget in the KV Cache with a Description-Length Price

The key-value (KV) cache of a transformer grows linearly with context length, and at long context it takes more GPU memory than the model weights. We present RAZOR, a method for training a small gate that decides, when each token is written to the cache, whether to keep it. The base model stays frozen. The gate is trained to leave the model's output distribution unchanged, measured as KL divergence to the uncompressed model, while paying a price for every token it keeps. The price is updated by dual ascent until a target memory budget is met, so no penalty weight has to be tuned. Keep/drop decisions are binary during training, with straight-through gradients, and the gate therefore trains on the same cache it produces at inference. Trained for 39 minutes on one T4 GPU with documents of at most 1k tokens, the gate (1.7% of model parameters) lets Qwen3-1.7B evict 76% of its KV cache on RULER-4k while scoring 86.6 +/- 0.6 over five seeds, against 88.9 without compression and 86.5 for KVzip+, which requires a second pass over the context. With architecture and data held fixed, it outperforms a gate trained on KVzap's regression targets by 2.3-7.1 points. The same gate transfers to 16k-token RULER and 100k-token InfiniteBench, and lets Qwen3-4B read 123k tokens on a 16 GB GPU, where the uncompressed model runs out of memory at 61k; at 29k tokens, decoding is 2.9x faster. Finally, across four budgets the learned price matches the slope of the cache's rate-distortion curve, which gives a direct measure of what a unit of KV memory is worth to the model.