Do Self-Improving Coding Agents Beat Textbook Baselines? Evidence from a Darwin Gödel Machine Port and Two Published Systems

Self-improving agent harnesses edit their own code and keep the edits that score better on a benchmark. We test what such loops find against simpler baselines: standard engineering changes ("textbook moves"), one-shot redesigns by a language model, and placebo loops whose children are exact copies. In a pre-registered port of the Darwin Gödel Machine (DGM) to hard competitive-programming problems, the improvements the loops found were mostly standard engineering changes. A single textbook move, a model loop with an adequate output budget, recovers about nine tenths of the best loop's gain at less than half its cost per task, and four of five pre-registered one-shot redesigns find the same fix without evaluation. The best combinations of textbook moves still fall one to five points short of the best loop agent in raw scores, and the registered test, with its selection rule fixed before analysis, fails: the chosen combination trails by 4.7 points (95% interval 0.4 to 9.5) against a margin of three points. A literal reading of the registration, which selects and scores on the same problems, passes on raw scores but not after adjusting for each batch's seed. Under two other task models the results are mixed. Rebuilt and run with a newer model on SWE-bench Verified tasks the search never saw, the DGM's released best agent resolves 43.1% of them in the DGM's own harness, against 56.5% for its initial agent and 73.6% for that initial agent with three textbook moves, at a quarter of the best agent's cost. About half of the best agent's runs end before a patch is recorded; with their patches kept, it does not differ significantly from its initial agent and still trails the textbook agent. ADAS's discovered agents lose to matched chain-of-thought sampling on two of four benchmarks with a newer model, tie on one and win on one. Claims of self-improvement should be measured against textbook moves, one-shot redesigns, placebo loops, unseen tasks and newer models. Code, both pre-registrations, the change log and per-task results are available at https://doi.org/10.5281/zenodo.23643515.