Adversarial audit reveals nine corrections and four inferential failure modes across ten preregistered preprints, highlighting that mechanical verification protects arithmetic rather than inference.
A research program that preregisters its protocols publicly, runs each once, and re-verifies every quoted number mechanically can still be wrong in ways none of those safeguards detect. We report what happened when this program — ten preprints on perturbational complexity, twenty-seven sealed protocols, all with passing mechanical verifiers — added one step: before depositing, each protocol was handed to an adversarially instructed reviewer whose default verdict was broken and whose task was to find (a) a way the sealed criterion could be satisfied with the hypothesis false, and (b) a way the protocol could never refute. Every finding was then recomputed from archived data before being accepted. The audit produced nine corrections across nine preprints, three of which change a conclusion. It also produced four structural findings we consider more useful than the corrections themselves. First, criteria were being tested against noise but never against trivial predictors: one paper's level criterion is passed by a constant that ignores the data entirely, and its scalar comparator lost through a truncation artifact rather than through the mechanism claimed. Second, mechanical verifiers checked that numbers were computed correctly but never that the sentences around them were supported — four papers asserted more than their own archived results allow. Third, the program's central estimator had never been read against a known truth in eight months of use; when a known-truth bench was finally built it passed (recovering injected dimensionalities of 3, 5 and 10 as 3.13, 5.11 and 10.10), but the bench also revealed a positive rectification floor and a monotone under-reading at low signal-to-noise that no paper had reported. Fourth, one paper's central contrast is quantitatively accounted for by a confounder the same program had already sealed and published in a separate protocol, without that finding ever propagating to the paper. We report the four gaps as four rules, each named with the failure that paid for it, and we report what the audit did not find: across twenty-seven protocols, no confirmatory dataset predates its seal; across eighteen confirmatory runs, no two share a single seed; and the simulation engine reproduces its analytic ground truth to five decimal places. The distinction that organizes all of this is between verifying arithmetic and verifying inference. Preregistration and mechanical verification protect the first. Nothing in our workflow protected the second until we built an adversary.
No takes yet. Share an insight, caveat, or question.
Nicolás Federico Galindez (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: