A Rate-Distortion Function for Model Merging

Merging LoRA adapters into one deployable model is judged empirically: methods are comparedon benchmarks with no notion of how much room is left to improve, and a method that wins on onebase model can become the worst on another with no warning. We address both problems by treatingmerging as lossy compression: how well can T adapters be represented by a single model, judged bythe task that suffers most? This rate-distortion view yields a closed-form lower bound on the achievableworst-task loss (an irreducible floor plus a compression cost) and an explicit encoder that matches it upto a logarithmic factor in T . Two practical tools follow. On real adapters the exact encoder fails for ameasurable spectral reason, but a one-line ridge correction repairs it; the repaired method, RD-ENCODERRIDGE, attains the lowest worst-task loss of the ten methods we benchmark, on all four base models, atdefault hyperparameters, with a lead that widens as more tasks are merged. The same analysis gives aCPU-only, minutes-long audit of trained adapters that predicts when the popular TIES method flips frombest to worst; we pre-registered its prediction for a third base (Mistral-7B) before the experiment ran,and the prediction held. On every adapter collection we measure, the floor is zero: the loss mergingadds is avoidable in principle, and our encoder recovers a substantial fraction of it without retraining(0.22 → 0.094 on Llama-3.1, 3-seed mean). Loss rankings also predict downstream accuracy rankingsin seven of eight model-metric pairs.