A Rate-Distortion Function for Model Merging

Merging LoRA adapters into a single deployable model is judged em-pirically: methods are compared on benchmarks with no notion of howmuch room is left to improve, and methods that work on one base some-times invert to worst-of-the-matrix on another with no warning. Weattack both problems with a single tool. A worst-task rate-distortionanalysis of LoRA merging produces a closed-form lower bound on per-coordinate NLL excess, Φ⋆ ≥ B2(1−deff /(T r)) in the data-free quadratic-loss limit, that admits an exact-construction encoder; on real adaptersthe exact construction fails for a measured spectral reason, but a one-lineTikhonov ridge salvages it into rd-encoder ridge — a merge methodwith the lowest worst-task excess, of all ten methods we benchmark attheir default hyperparameters, on all four bases at T = 4 (including theclosest structural relative, data-free RegMean). When the two closestcompetitors are themselves tuned by their key hyperparameter (TIES/DAREdensity, RegMean ridge λ), rd-encoder ridge stays best on three of fourbases; on the saturated Yi-1.5-9B-Chat base a tuned RegMean edges it(0.033 vs 0.037 nats) — the one base where we independently flag NLLexcess as an unreliable proxy. On a separate T -scaling task pool it holdsa lead over Task Arithmetic that widens with task count (excess ratio0.84 → 0.68 → 0.52 at T = 2, 4, 7 on Llama-3.1-Instruct). The sameanalysis yields a 1-CPU-minute audit of any trained adapter cohort thatdiagnoses when classical methods such as TIES invert from best to worst:a perturbation derivation of the TIES sign-election update predicts an am-biguity window R⋆ ∈ [0.025, 0.075] for the per-task win-share range: aboveit TIES inverts to worst, inside it TIES is ambiguous. One base above thewindow (Yi-Chat, R = 0.077) inverts as predicted; two inside it — Llama-Instruct (R = 0.028) and a pre-registered Mistral-7B-v0.3 (R = 0.0326)— are ambiguous as predicted, TIES neither inverting nor winning (onMistral, second of six, 0.092 nats below Task-Arithmetic). We are deliber-ately cautious here: the window’s constants were fit to the two in-samplepoints, the below-window regime (TIES wins outright) is untested, andthe one out-of-sample test confirmed only the low-information ambiguousprediction — so we present the threshold as suggestive, not validated.Across 4 bases × 2 downstream metrics, NLL ranking predicts accuracyranking on 7 of 8 cells (each Spearman ρ is over only five methods andindividually low-powered; the across-cell consistency, not any single coef-ficient, carries the signal). Every cohort we measured sits in the floor-zeroregime, so the bound’s floor is identically zero and its rate term falls be-low sub-bit resolution on real merges: the theory’s empirical role is theregime diagnosis — none of the 0.10–0.22 nats of worst-task excess is aninformation-theoretic floor — not a numerical prediction. How much ofthat excess is algorithmically recoverable is open; rd-encoder ridge re-covers a measurable fraction without retraining (0.22 → 0.094 on Llama,3-seed mean). The headline for practitioners is the five-rule decision treeof §6.8.