The Multiple-Choice Trap: Conservation, Cross-Rig Variance, and the Limits of Boosting Small Legal Language Models

We document a comprehensive empirical investigation into boosting small language models for legal multiple-choice reasoning under deployment constraints (sub-3 GB inference footprint). Working with the Qwen 2.5 1.5B-Instruct base, we run 8 distinct training-side variants spanning self-distillation rounds, hard-example weighted training, EWC regularization, and continuation training, plus 8 LoRA-adapter merge configurations on top of those variants—16 trained configurations in total, all evaluated on a held-fixed 200-question Multistate Bar Examination (MBE) benchmark. We report three findings. First, we observe a conservation pattern: across all training-side variants the aggregate accuracy stays within a ~4 percentage point band (29.0%–33.5% on canonical evaluation), while per-domain accuracies redistribute substantially. Combining adapters via 8 merge configurations does not break this pattern. A parallel attempt on Qwen 2.5 0.5B-Instruct produced a net regression below baseline and two catastrophic GRPO collapses, suggesting the conservation pattern has a scale-dependent floor below which training actively hurts. Second, we document an adapter-specific cross-rig variance: the identical adapter on the identical questions can yield MBE scores 4–8 percentage points apart across attention backends (vLLM with FlashAttention 2 vs. HuggingFace SDPA), in ways that can exceed measured training gains. Third, inference-time methods (domain-routed sampling combined with retrieval-augmented generation against a curated legal-doctrine corpus) lift the deployable 1.5B system to 43.0% MBE accuracy (robust) and 45.0% (with per-domain method selection), a +13.5–15.5 percentage point gain over baseline—but qualitative evaluation on 30 open-ended legal questions shows the same model confidently reverses the holding of Erie Railroad Co. v. Tompkins. We argue this is the multiple-choice trap: multiple-choice fine-tuning at this scale teaches answer-pattern recognition rather than substantive doctrine, and deployment claims for small expert models therefore require open-ended validation alongside benchmark scores.