Who Writes the Answer Key? Lessons from Forty-Eight Rounds of Trying to Benchmark Autonomous Physics Research

We report a month-long attempt (23 September to 21 October 2026) to build an automatically scored benchmark of autonomous scientific research in physics that models could not saturate without genuinely improving. The project was conceived and led by one human, who set the goal, approved every model run, carried every message between three AI systems (ChatGPT Pro as designer, Codex as builder and runner, and Claude as director and verifier), and supplied several of the ideas the designs were built on. Over about 48 rounds the team built and tested many designs:- toy discovery tasks and constructive research with a checker;- generated worlds with disguised scientific revolutions, twin laws and hidden unifications;- calibrated far forecasts;- clue worlds;- multi-layer discovery horizons;- physics truths that need large computation;- method invention scored exactly. In 162 subject-model episodes, 123 of them with gpt-6.1-sol, that model solved nearly every single-principle discovery world we scored (23 of the 49 scored cells supplied the law or a hint, or were controls). Its one systematic failure, far forecasts without the candidate law family, was not shown to be specific to the model: four generic public-information baselines, tested after the fact, did worse overall. Every design was eventually rejected. Read in hindsight, many rejections fall into four overlapping groups by how the answer key was built, although some had more local causes (a reference that did not converge, a design assumption that failed, a clue printed on the task card, prior art). Keys built from the designers' knowledge were reached by the solver; a trace audit could not tell whether by recall or by fitting candidate families. Authored laws were found by the model, or their scored behavior was learned by generic programs. Computed truths could support only a small separation on one machine, and we derive a cap on that separation. In the verifier families, standard methods reached certified optima at small sizes; in the last family, repeated runs of one method agreed without an optimality certificate, and we closed it by decision. Producing answer keys was easy; we did not establish an in-house task family that combined validated scoring, attainability from the supplied evidence, resistance to the declared shortcuts, and useful headroom for the tested model. We state this as an empirical conjecture about design teams with resources like ours, with a test that would count against it, and we place the known constructions that look like exceptions. Outside sources of keys (researchers' unpublished results, nature, much larger compute) are worth exploring, but our record establishes neither that outside provenance is necessary nor that it is sufficient. We state three rules for benchmark builders: programs reject and models certify; a speed gap is not research headroom; measure the gap at the scored precision. We document the AI designers' own failures, several of which resemble capabilities the benchmark sought to measure, and sketch a protocol for outside answer keys together with its failure modes.