Existing online KV cache quantization methods optimise the compression primitive — uniform 2-bit (KIVI), outlier-aware fixed precision (KVQuant), random orthogonal rotation (TurboQuant) — and assign one format to all blocks at thepopulation level. We present PalQuant, an adaptive system that optimises the compression architecture: at inference time, each (layer, head, chunk) selects 4 active formats from a fixed 16-format codebook spanning 0.25–16 bits per element, with each block dynamically routed to one of the 4 based on its measured reconstruction error. Selection uses structurally distinct K and V error metrics matching their attention paths: mean-of-top-4 weighted absolute error for K (capturing softmax-amplified outliers), warp-mean squared error for V (the L2 budget for V’s attention-weighted contribution), with parameter-free sink protection via tanh-weighted Q·K alignment statistics. Deployed identically across three models with no per-model calibration data, PalQuant achieves 7.42× cache compression on Qwen3-8B, 7.04× on Qwen3 30B-A3B (MoE), and 5.02× on Llama-3.2-3B, validated end-to-end by a multi session story rewrite test where each concurrent session must preserve a distinct assigned character identity (name and gender) under aggressive compression. PalQuant passes the multi-session test at all 11 compression levels (C0C10) up to 7.42× CR, while uniform Q4 0 fails at 3.56× CR on Llama-3.2-3B and Qwen3-30B-A3B — even though uniform Q4 0 holds competitive bulk PPL, demonstrating a structural dissociation between perplexity and the multi-session quality metric (§4.5). The selection mechanism’s decisions transfer cleanly across MoE/dense, model families, and model sizes because pre-RoPE K/V activation structure is universal at the per-block level. PalQuant approaches Kitty’s offline calibrated compression band (7.42× vs ˜8×) without calibration data and substantially exceeds TurboQuant (4.6× at FP16-equivalent quality). Native CUDA kernels on a single RTX 4090 Mobile (16 GB) support up to 256/120/64 concurrent Llama-3.2-3B/Qwen3-30B-A3B/Qwen3-8B sessions at 168K/75K/41K totalKVcache tokens.
Johnathan Paul Sharratt (Sat,) studied this question.