Answers are grounded in this paper and cite their sources.
Uncovers how the additive input pathway undermines state tracking in neural networks, suggesting insights for automata learning.
Linear RNNs with input-dependent Householder-product transitions (DeltaNet/DeltaProduct-class) can provably represent hard state-tracking automata, yet trained models fail to length-generalize -- a gap recent work attributes to optimization, without a causal account. We give one, in a pre-registered, within-architecture causal ablation: the same model with one term deleted -- the additive input injection bₜ = Wb eₜ. With bₜ, models fit length 32 and collapse out-of-distribution on parity, S₄, A₅, and non-solvable S₅ word problems (0.20 at position 512 on S₅). Without it -- input acting only through the orthogonal transitions -- the same architecture learns the exact automaton: median accuracy 1.00 at 16x the training length, at every width admitted by a representation law we state and test: the minimal number of Householder factors per token equals the maximal reflection length of the task's generators in the format-pinned representation (parity 1, S₄ 3, A₅ and S₅ 4); below it, nothing fits. The contrast with DeltaProduct's S₄/A₅ at nₕ=2 (group-element classification, $SO(3)$ realization) shows the law is representation-relative: task format is a hidden variable in state-tracking benchmarks. Two pre-registered arms locate the mechanism. (i) Initialized at a verified-exact solution with Wb=0, Adam grows the additive path and pulls the model off the exact solution; $-b$ controls stay at 1.00. (ii) Our registered prediction that the fit routes through bₜ fired its kill criterion: all 49 fitting seeds retain in-domain fit under Wb:=0 -- and at law-minimal width, zeroing Wb at inference restores exact generalization (parity 5/5, S₅ 5/5, S₄ 4/5, A₅ 4/5). The additive pathway is parasitic: it destabilizes, then conceals, a correctly learned automaton. All 202 runs pre-registered; all numbers regenerate from artifacts. Code, pre-registration, and all 202 run artifacts are included in this deposit.
No takes yet. Share an insight, caveat, or question.
Gunner Levi Howe (2026) studied this question.