Answers are grounded in this paper and cite their sources.
Item response theory lets us ask whether a benchmark item behaves differently for two groups of language models that have the same overall ability, which is the classical definition of differential item functioning (DIF). Model panels, however, are not samples of independent examinees: a base checkpoint and its instruction-tuned twin answer most items identically, and several sizes of one family share data and tokenizer. We show that this dependence breaks the standard permutation tests. On synthetic null panels shaped like ours, a modellevel label shuffle rejects a true null in 0.58 to 0.62 of cases at α = 0.05, and a leave-familyout variant that compares a held-out family against in-sample residuals rejects in 0.83 to 0.86; tests that permute whole units (twin pairs) or refit the null hold close to the nominal level. Re-auditing 29 open models (135M to 3B parameters, 12 families) on ARC-Challenge, ARC-Easy, OpenBookQA and MMLU with valid tests changes the picture we reported in an earlier draft. Family-versus-rest DIF tests that were highly significant under the naive tests (Qwen2.5, SmolLM2, Falcon3, TinyLlama) no longer survive Holm correction under any valid test, and two families (OLMo, TinyLlama) turn out to be untestable because their only evidence was a twin pair. Lineage structure is nonetheless real: ability-controlled residuals of two models from the same family but different checkpoints correlate more than residuals of models from different families (pooled Δr = 0.16, 95% CI [0.13, 0.19], permutation p ≤ 0.0001), also among base models only, and a held-out model's family is recovered with its twin removed from the reference set in 18 of 20 cases (permutation null mean 0.09). Two cautions qualify this. Raw correctness vectors recover family almost as well (0.80 versus 0.90), so ability control is not what makes attribution work, and cross-family pairs that share a tokenizer are about as similar as same-family pairs, so part of what looks like lineage may be tokenization under log-likelihood scoring. A paired within-twin test also overturns our earlier null for instruction tuning: tuning appears to leave a modest item-level signature (pooled exact p = 0.002; p = 0.004 after removing smooth person misfit), although part of it reflects instruct models fitting the IRT model less well. Rankings are stable when high-DIF items are removed. Finally, an IRT-residual contamination detector beats label-only baselines in controlled LoRA injections but is dominated by Min-K% Prob whenever token probabilities are available, so we do not recommend it. We release the tests as a small open tool.
No takes yet. Share an insight, caveat, or question.
Sharven Rane (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: