Same Label, Different Experiment: Measuring the Prompt-Injection and Jailbreak Defense Evaluation Record

Practitioners choose prompt-injection and jailbreak defenses by comparing attack-success numbers across papers. We treat the published record as the object of measurement and ask what comparisons it supports. Under a protocol committed before the sweep and anchored in Bitcoin before any value was extracted from the swept corpus, we screened 3,865 candidate papers, included 1,610, and extracted 62,835 reported values from 667 papers. A script located 91.7% of them on their source lines, and only those enter the analyses. Most defenses have no evaluation from outside their authors' group: 70.6% to 84.1% of 831 named defenses, of which 465 were proposed in 2026. In the oldest large cohort, 52.6% (interval 40.9 to 63.1) remain so two years after proposal. Among cross-paper pairs that meet the protocol's matching rule, 53.3% differ by a factor of two or more (interval 10.5 to 75.0, 13 pairs of papers). An audit of 100 label-matched pairs by the first author, blind to the extracted conditions but not to the values, finds 11 that are the same experiment, all from 2 pairs of papers. In that sample the judge that scores attack success was the most common hidden difference. Orderings under adaptive attack cannot be shown to carry across papers: 12 of 76 comparisons disagree, with a paper-clustered interval of 0.0 to 60.0%. Excluding the 30 pilot papers, which include the main adaptive-attack studies and the papers proposing the best-known in-model defenses, no in-model defense has a protocol-matched pair of its own number and an outside adaptive one. With them, 1 does, too few to test. We report each pre-registered prediction with its verdict (a 30-paper pilot informed three of them, so each is also given without the pilot), every deviation from the protocol, and a reporting protocol aimed at the failures we measured.