What a Local Screen Could Not Tell Us: Measuring a Small Offline Coding Agent Under a Twelve-Hour Budget

In the Gemma 4 Developer Agent competition the model, the tools and the scorer are fixed, and an entry is only an agent configuration: its prompt, its limits and its sampling settings. Entrants choose between configurations the way most agent builders do, by running them on a small public task set before spending a scarce hidden submission. We report what that practice told us over four weeks, from 20 complete runs on the 74 public tasks across two local setups (the scorer's own four-L4 hardware and a single GPU paced to it) and 11 configurations submitted to the hidden set. The local screen was a poor guide. On the scorer's hardware, a change that beat its parent on nine tasks and lost none scored lower on the hidden set. On the paced GPU, three configurations that each solved 28 of 74 locally scored 0.10, 0.10 and 0.17, and the rank correlation between local and hidden scores was +0.49 (exact permutation p = 0.32, seven configurations). Three measurements explain much of this. Between 91 and 97 percent of each task's wall time is the model generating text, so the per-task time cap is a token budget. Solves arrive late, so longer caps buy local solves that a hard twelve-hour limit on the whole run cannot pay for: a public configuration that scored 0.24 for its author ran out of time on both of our attempts. And prompt rules that each fixed one task lost seven tasks when merged. We also ran Google's Harness² with Gemma as its own harness improver: 23 of 30 improver sessions overflowed the model's 32k context and no change to the harness was adopted. Every reading rule was committed to git before its result, every number is regenerated from saved files, and all code and data are public. Code, configurations, per-task results, traces and the append-only reading-rule log: https://github.com/rodriguescarson/gemma-swe-agent (Apache-2.0).