The key-value (KV) cache in autoregressive transformers grows linearly with context length and dominates memory consumption at long contexts. Existing compression methods treat each cached token independently, ignoring the substantial temporal redundancy between consecutive cache states within a layer. We propose TurboMirage, an adaptive predictive codec that treats KV cache compression as a temporal coding problem, drawing on the design principles of H.264/H.265 video compression. TurboMirage partitions the KV stream into adaptive Groups of Pictures (GOPs) via cosine-similarity gating, predicts P-frame tokens using a layer-conditioned low-rank predictor with learned embeddings, and compresses the prediction residuals to INT4. The predictor is initialized as a perfect COPY (zero residual correction) and trained online via distillation on live KV traces, requiring no calibration data and no modification to model weights. All codec operations are implemented as vectorized GPU kernels that JIT-compile into fused shaders. We evaluate TurboMirage across multiple large-scale transformer architectures ranging up to 35B parameters, including standard Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and Mixture-of-Experts (MoE) formulations. The learned predictor achieves near-optimal distortion rates, yielding up to 2–3 orders of magnitude lower reconstruction MSE than static baselines, effectively resolving quantizer overflow phenomena observed in larger models. With vectorized kernels, TurboMirage retains 96% of baseline generation throughput. Codec-active evaluation on downstream tasks demonstrates bit-identical accuracy on evaluated subsets, confirming that the codec reliably operates below the model’s decision boundary
Gaurav Saini (Mon,) studied this question.