Reach, Not Judgement: What Nine Preregistered Studies Show About When Self-Modification Pays Off

Definitions of recursive self-improvement ask that a system rewrite its own improvement mechanism; they rarely ask whether the rewriting helps, or whether what it helps with generalises. We report nine preregistered studies of one bounded self-improving program-repair system, each locked and hash-frozen before its confirmatory run. A writable generator of self-modifications beat a fixed one (+5.2 percentage points, pp) and a self-extending vocabulary beat the designer's (+4.1 pp), though a complete designer-supplied vocabulary beat both. The system met every clause of a published Level-4 definition, yet five successive tests found no causal benefit of rewriting the improvement mechanism, and the last found harm (-1.12 pp). Layer-by-layer ceilings explained why: the rewritable parameters had about 1.7 pp to give, while the headroom lay in the policy's form, observations and search. When the mechanism's structure became rewritable, one of two campaigns adopted a structural change on its own evidence, and the preregistered causal criteria were met for the first time (+2.57 pp versus the default, p = 0.0028; +3.52 pp versus random rewriting, p = 0.0011). Across studies, the improvement loop's products transferred to 164 real bugs (+4 to +5 bugs over the starting policy, replicated three times), but no rewrite of the mechanism added a significant real-bug gain, and gains on training families largely vanished on held-out families. A post-hoc analysis traces this to the system's observations: no policy form transferred more than about 5 pp to unseen families. Productive self-modification here was limited by what the system could change, not by how well it judged its changes, and its benefits stayed in-distribution.