A Green Prompt Suite Is Not Evidence of a Better Prompt

Prompts are now versioned artifacts, and the natural next step is to defend them with a regression suite in continuous integration. Such a suite asserts properties of a single model reply: that it matches a declared output contract, that it solves a reference input, that a judge model scores it well. The system the prompt serves is not a single reply. It is a multi-turn, execution-grounded loop. We ask whether the first predicts the second. We run five prompt variants, a competent baseline and four concrete edits each motivated by a documented prompt-engineering practice, through an execution-grounded repair loop over 184 defective programs from two public corpora and two models, for 1,840 episodes, holding the parser, the diagnostic channel, the turn budget, the temperature and the seed fixed so that only the prompt varies. Outcomes are graded on a hidden test suite the model never sees. The design and a falsification criterion were fixed in advance, and every amendment to the analysis is listed. The two instruments disagree in magnitude, by roughly ten- to twenty-fourfold on the contract assertions and, on the closest like-for-like comparison, by elevenfold on one model and under twofold on the other. Across the same four prompts on G4-8B, contract compliance spans 100.0 points, the entire available range, and single-turn pass rate spans 53.3 points, while deployed resolution spans 4.2 points. Of 48 variant-pair comparisons, 27 are cases where only the CI side excludes zero. Two independent checks then find no deployed signal these data can resolve: across the 12 distinct deployed comparisons, 0 survive Holm correction, while the same correction applied to the CI side leaves 24 of 27 standing, and re-running one model under a second seed moves each variant's deployed rate by up to 9.7 points, more than the gaps separating the variants. We therefore make no claim that any prompt deploys better than any other. The claim is the asymmetry, a post-hoc statistic written after that replication and measured on the replicated model against its seed replication on both sides: the two contract assertions vary between prompts by two to fifty times their own seed-to-seed movement, and the single-turn pass rate and the deployed outcome do not, a split that survives a robust recomputation. A post-hoc control edit identical to the baseline on turn one, scored at exactly zero by every assertion, still changed one model's visible-suite resolution under correction, though not the hidden suite we treat as primary. Behind the contract half of that split, the asserted contract is stricter than the deployed parser, so for one variant 90.2% of turn-1 replies violate the contract and are executed by the harness anyway. We do not identify a mechanism for the pass-rate or judge disparities. The usable consequence is a sizing rule. A 20-task suite, generous for something run on every commit, estimates a deployed difference only to within about 15.9 points, and 8.1 even on the most favourable pair, while the deployed rates here span 4.2 to 6.9 points: such a suite cannot separate differences of that size from zero, and a confident verdict conveys none of the imprecision. Size the suite against the change worth blocking a merge over, and assert what your deployed parser will reject rather than what you wrote. We also looked for a cheap gate and did not find one: four candidates, none shown to beat a coin, the best still costing 83.8% of the full loop's tokens. That comparison scores each gate against a deployed sign the replication shows is not reproducible, so we report it as a null instrument rather than evidence against cheap gates. We release the corpus builder, the loop, every prompt, the full episode log, and the analysis that produces every number here.