Coding agents usually stop when available tests pass, even when those tests omit stated requirements. We study this failure in a cloud-architect/local-builder arrangement with gemma4:12b repairing small Python modules on a consumer laptop GPU. A cloud reviewer receives only signatures, docstrings and data-type fields, never function bodies, and returns contract tests or a checklist. A separate hidden evaluation scores the resulting code. Across 50 task instances and 150 paired runs with corrected measurement, the contract-test arm reduced silent false successes from 37 to 4 and raised verified completion from 66.0% to 87.3%; task-level silent failures fell on 12 instances and rose on none (p=0.0005, post hoc). These are pooled, partly development-reused tasks, not 50 independent new specifications. In a pre-registered campaign on 25 new tasks, both review forms raised verified completion from 58/75 to 71/75 and reduced silent false successes from 9/75 to 1/75. The checklist matched the aggregate verified-completion rate of contract tests; equivalence was not tested. Including one generation per task reused across three seeds, campaign time increased by 11.8% for the checklist and 53.7% for tests. The registered task-level test did not reach significance (p=0.125); the contract-test gains were concentrated in multi-file tasks. An exploratory disclosure ablation points to the importance of docstrings. The results support specification review in this bounded setting, while leaving its causal components and performance on real repository defects unresolved.
No takes yet. Share an insight, caveat, or question.
Pylypenko Oleksandr (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: