Demonstrates the importance of retrievable specifications for autonomous coding performance, indicating that the executor model benefits significantly from a held spec.
Does autonomous coding improve mainly by adding agents, or by giving a model a place to keep knowledge? We separate these with two controls: the executor model is frozen (so any gain is the substrate's, not the model's), and all tasks target APIs invented after the training cutoff (so success cannot be memorization). Under greedy decoding, a frozen 4B-parameter model on an invented deterministic physics world it must implement rises from 14% to 86% once the substrate holds the spec — a task impossible cold because every step needs an arbitrary constant the model cannot know. The same lift appears on a stateless suite (40→100%) and an eight-feature stateful application (~42→~96%). Three findings sharpen the claim: the one task that does not flip with the spec is its reasoning-bound integration loop, so a single suite shows the substrate rescuing every knowledge-bound subtask and none of the reasoning-bound one; storing a capable model's one-time verified solution then rescues even that reasoning-bound task, turning reasoning into retrieval; and agents teaching each other plateau ~25 points below a held spec, stuck on exactly the unguessable-contract features in every seed. The deeper thesis: some task-critical information can live only in a persistent, retrievable store — not in weights, prompts, or transient agent conversations.
No takes yet. Share an insight, caveat, or question.
Pranab Sarkar (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: