Coding agents that fail to recover from a broken build are usually explained by the model: a weaker model gets stuck more often. We report a controlled experiment showing that, with the model held fixed, a large part of that failure is supplied instead by the development environment, specifically by what the toolchain is willing to report after a failed build. We inject a single known compile fault into a real repository inside the benchmark's own container, hold the model, the task, the step budget, the command set, and the observation parser fixed, and vary only the rendered diagnostic. Collapsing the diagnostic to "build failed" raises the stall rate, the fraction of episodes that never restore a clean build, from 0.48 to 0.85 in Rust and from 0.40 to 0.92 in Go, with instance-clustered intervals on the difference that exclude zero in both ([+0.25, +0.48] and [+0.40, +0.63]). A length-matched placebo that preserves the file:line:col pointer but destroys the explanation isolates one component of the diagnostic at fixed output length: it tracks the intact diagnostic in Rust (0.53) and sits between the extremes in Go (0.63). Holding output length fixed and deleting only the explanation leaves the stall rate unchanged in Rust (p = 0.58) and raises it in Go (p = 0.01), so how much the explanation adds appears to depend on the environment, a pattern our sample is underpowered to confirm as a formal interaction (p = 0.22). A fourth condition, run separately under a later harness build with its own matched control, keeps the length and masks only the file:line:col coordinates: removing the address alone raises stalling by 0.32 in Rust and 0.48 in Go. The effect replicates across a mid-tier open-weights model and a frontier model, which recovers from every fault under both the intact diagnostic and the length-matched placebo, yet stalls on 0.57 of episodes once the observation collapses to a bare failure signal. Go is the strongest case: its native compiler output is already about as terse as our degraded Rust condition, and removing it still more than doubles stalling. A re-run of the three Rust arms under a later build reproduces the direction at a smaller magnitude, so we read every effect size here as a single-run estimate. Agent reliability is therefore partly an integration property that harness and toolchain designers control directly, and it is measurable for under five dollars of inference on a laptop.
No takes yet. Share an insight, caveat, or question.
Simarjot Khanna (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: