Finds LoRA knowledge distillation recovers reasoning in quantized models, suggesting effective training strategies.
Post-training K-trit-plane quantization of LLM weights to {-1,0,+1} is near-lossless on average but collapses multi-step reasoning at K=2 (≈3.3 bpw) on small models—GSM8K falls to near zero while knowledge survives. We ask whether a cheap, short correction can buy the reasoning back, and study a LoRA knowledge-distillation (LoRA-KD) path: a small low-rank adapter on the frozen ternary backbone, trained with logit distillation from the full-precision (FP) teacher and initialized from the quantization-error SVD (LoftQ-style). Findings, each labelled with its scope. (1) LoRA-KD rescues the K=2 collapse: GSM8K goes from ≈0 to 0.05–0.42 (3-seed means; six models across three families; the 0.05 low end, Qwen2.5-3B, is seed-unstable; Falcon3-3B is the exception that does not collapse at K=2). (2) The same recipe brings the recommended K=3 point close to FP: K=3+distill reaches FP on the larger models (2/6 exactly, within ≈0.06 on the other larger models), but the 0.5B stays ≈0.10 below—so we avoid the unqualified "lossless". (3) But it is not a shortcut: plain K=3-PTQ beats K=2+distill on all six models (sign-consistent, p≈0.016), so spending the bits on a third plane dominates spending them on a trained correction. (4) The recovery is mostly general, not task-format memorization: distilling on a generic corpus (WikiText, no math) already recovers the bulk of the distillation gain on GSM8K (0.20 of the task-aligned 0.31), the task-aligned data adding the rest—the logit-KD repairs the quantization damage at the distribution level. (5) Negatives and a scaling correction: a single plane (K=1) is unrecoverable; a controlled A/B shows the LoRA scaling α/r—not the rank—governs recovery (rank-16 at α=2r matches rank-64; α=128,r=16 collapses to 0.007), so a rank-16 adapter (≈5% weight memory) suffices, though we report the rank-64 flagship (≈19%) as the conservative operating point. We also separate the white-box recipe (teacher logits) from a black-box / API variant (cross-entropy on a teacher's responses), and find the API variant viable—it recovers ≈75% of the white-box gain with no teacher weights or logits. The results are preliminary (a handful of models, GSM8K-centric); the robust objects are the orderings, not the magnitudes.
No takes yet. Share an insight, caveat, or question.
Adriel Domingues de Souza Andrade (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: