Anchor-Token Masking for Out-of-Distribution Generalization in Masked-Diffusion Language Models

AbstractWe empirically compare three masking strategies for fine-tuning masked-diffusion language models(MDLMs) on retrieval-grounded data: anchor-token masking (prefix question never masked, answerstatement always subject to noise schedule), random-position masking (uniform over both prefix andtarget), and a reverse-asymmetric control (prefix always subject to schedule, target never masked).Using rank-16 LoRA adapters trained on a 1.3B-parameter undertrained MDLM (Cassandra T1,epoch-5 cross-entropy 2.26) for 300 steps on 144 (Q, A) examples, we evaluate corpus-vocabularyoverlap of generated text around forced-anchor positions across in-domain queries and four held-outsemantic domains spanning astronomy, medicine, law, and culinary arts. Anchor-token maskingproduces a 1.67× pooled out-of-distribution generalization advantage over random-position masking(95% bootstrap confidence interval [1.51×, 1.85×]; P(anchor > random) = 1.000 across N = 48 heldout queries), with the OOD advantage statistically larger than the in-domain advantage (CI [1.18×,1.34×]; non-overlapping with the OOD CI). The reverse-asymmetric control performs significantlyworse than random-position masking on held-out queries (diff = −0.038, P = 0.006), establishingthat the direction of the asymmetry is load-bearing, not merely its presence. A sample-efficiencysweep across six (data-size × step-count) configurations shows the advantage peaks at low trainingbudgets (1.97-2.19× at 100 steps) and attenuates monotonically with step count (collapsing to 1.07×at 1000 steps on 144 examples), characterizing the technique as a low-shot fine-tuning advantagerather than a fundamental scaling advantage. Four independent mechanism probes — direction-ofasymmetry, attention-pattern instrumentation, LoRA weight allocation analysis, and a direct V/Oablation — establish V/O (value/output) projections as causally necessary and sufficient for theinductive bias: V/O-only LoRA (with Q/K frozen) outperforms full LoRA at half the trainableparameters (0.416 vs 0.337 held-out corpus_overlap ), while Q/K-only LoRA falls below even therandom-mask control (0.136 vs 0.214). The mechanism is therefore not “the model learns to look atanchors” but “the model learns to extract anchor-extending content from positions it was alreadyattending to.” We discuss implications for retrieval-augmented MDLM design, fine-tuning APIspecification, and vertical productization. All artifacts are released under Apache 2.0.