AbstractWe empirically compare three masking strategies for fine-tuning masked-diffusion language models (MDLMs) on retrieval-grounded data: anchor-token masking (prefix question never masked, answerstatement always subject to noise schedule), random-position masking (uniform over both prefix andtarget), and a reverse-asymmetric control (prefix always subject to schedule, target never masked). Using rank-16 LoRA adapters trained on a 1. 3B-parameter undertrained MDLM (Cassandra T1, epoch-5 cross-entropy 2. 26) for 300 steps on 144 (Q, A) examples, we evaluate corpus-vocabularyoverlap of generated text around forced-anchor positions across in-domain queries and four held-outsemantic domains spanning astronomy, medicine, law, and culinary arts. Anchor-token maskingproduces a 1. 67× pooled out-of-distribution generalization advantage over random-position masking (95% bootstrap confidence interval 1. 51×, 1. 85×; P (anchor > random) = 1. 000 across N = 48 heldout queries), with the OOD advantage statistically larger than the in-domain advantage (CI 1. 18×, 1. 34×; non-overlapping with the OOD CI). The reverse-asymmetric control performs significantlyworse than random-position masking on held-out queries (diff = −0. 038, P = 0. 006), establishingthat the direction of the asymmetry is load-bearing, not merely its presence. A sample-efficiencysweep across six (data-size × step-count) configurations shows the advantage peaks at low trainingbudgets (1. 97-2. 19× at 100 steps) and attenuates monotonically with step count (collapsing to 1. 07×at 1000 steps on 144 examples), characterizing the technique as a low-shot fine-tuning advantagerather than a fundamental scaling advantage. Four independent mechanism probes — direction-ofasymmetry, attention-pattern instrumentation, LoRA weight allocation analysis, and a direct V/Oablation — establish V/O (value/output) projections as causally necessary and sufficient for theinductive bias: V/O-only LoRA (with Q/K frozen) outperforms full LoRA at half the trainableparameters (0. 416 vs 0. 337 held-out corpusₒverlap), while Q/K-only LoRA falls below even therandom-mask control (0. 136 vs 0. 214). The mechanism is therefore not “the model learns to look atanchors” but “the model learns to extract anchor-extending content from positions it was alreadyattending to. ” We discuss implications for retrieval-augmented MDLM design, fine-tuning APIspecification, and vertical productization. All artifacts are released under Apache 2. 0.
Garren et al. (Mon,) studied this question.