Answers are grounded in this paper and cite their sources.
In silico experiment reveals near-zero evidence-sensitive rule selection in standard transformers, indicating failures stem from training procedures rather than architecture.
High accuracy after anomalous evidence need not mean that a language model revised a rule: the same answer may already be its default. We separate rule access, default generalization, and evidence-sensitive selection in three synthetic microworlds with transformers of approximately 0.82M–4.79M parameters. A bidirectional test holds context inputs fixed while changing distance facts between two competing rules; answer permutations separately control the token multiset. Across 50 boundary-world checkpoints and ten context draws per query, group-median selection effects range from -0.002 to +0.013, despite high tag-given access in several arms. Given the two-rule hypothesis class, a symbolic selector identifies both rules perfectly. Three separately trained models of the same 0.82M architecture reach bidirectional accuracy 0.966–1.000 on 28 held-out direction queries after explicit episodic training. Corpus interventions explain a separate effect: full-range old-rule documents suppress alternative-rule defaults, whereas distance-only documents preserve direction accuracy. Completing a frozen low-dose experiment shows that 1–30 old-rule documents do not reproduce the collapse observed near 100 documents. These results localize the observed failure to evidence use under the tested training procedures, rather than information insufficiency or a demonstrated inability of the architecture. The contribution is a controlled diagnostic of rule selection and its training dependence, not a mechanism of scientific discovery.
No takes yet. Share an insight, caveat, or question.
Dekun Yang (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: