Answers are grounded in this paper and cite their sources.
AI agents are compared by scores, but what papers, leaderboards and model launches publish are claims: that a system is “#1”, better than a named rival, as good at lower cost, or on the cost–accuracy frontier. We propose claim-level evaluation: the unit of reporting is the claim, stated in one of a small set of testable forms and published with its bootstrap support proportion—the fraction of resampled datasets in which it holds. We give a test for each claim form and a one-command tool that applies it to any results file. To show why this is needed, we applied a protocol-bounded search to selected systems and sources, collecting 77 published claims from 51 public sources about systems on SWE-bench Verified, Terminal-Bench 2.0, τ²-Bench, and the Holistic Agent Leaderboard, and tested each against the per-task results it describes. Of the 43 claims about runs whose data are public, 29 (67%) fall below the 0.9 bootstrap-support threshold, and 6 are not even true of the published numbers. Rank claims fare worst: 2 of 21 “#1” claims are supported. This does not imply false reporting, but rather that benchmarks lack the statistical resolution to distinguish narrow margins: the unsupported leads have a median gap of only 0.4 points, requiring a median of 5.7× as many tasks to verify. Cost–accuracy frontier claims fare best (5 of 5), because costs differ by large factors. Seen at the claim level, SWE-bench Verified’s leader cannot be distinguished from 4 other entries and HAL’s τ-bench leader from 11; an unsupported lead would need a median of 5.7 times as many tasks to be supported. We release the claims, the coding rules, and the tool.
No takes yet. Share an insight, caveat, or question.
Harish Mashetty (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: