Magician: VPU-Accelerated KV Cache Disk Offloading for On-Device Large Language Model Inference

On-device LLM inference is constrained by the memory capacity and bandwidth required by growing key-value (K/V) caches. Magician investigates VPU-accelerated compression and disk offloading for K/V-cache data. It characterizes tensor distributions across models and tasks, develops lossless and controlled-lossy HEVC-compatible compression methods, and evaluates them using GPU servers and a mobile SoC testbed. Key metrics include compression ratio, reconstruction fidelity, model accuracy, encode/decode throughput, storage capacity, latency, and energy consumption. The project aims to determine when dedicated mobile video hardware can reduce CPU/GPU memory and I/O pressure without unacceptable quality degradation. Reproducibility is supported through versioned code, configurations, manifests, random seeds, software versions, and machine-readable results.