Adversarial audit reveals widespread inferential and computational errors across ten preregistered preprints, highlighting critical blind spots in automated verification.
A research program that preregisters its protocols publicly, runs each once, and re-verifies its quoted numbers mechanically can still be wrong in ways none of those safeguards detect. We report two audit rounds of such a program — ten preprints on perturbational complexity and reproducible response dimensionality, 31 publicly sealed protocols, seven of the ten preprints carrying a mechanical verifier at the time of the first round. Round one (31 August 2026) applied, retrospectively to ten already-deposited preprints, a procedure the program has since made mandatory before each deposit: each group of papers was handed to an adversarially instructed reviewer whose default verdict was broken and whose task was to find (a) a way the sealed criterion could be satisfied with the hypothesis false and (b) a way the protocol could never refute. Every finding was recomputed from archived data before being accepted. Counting one correction per distinct statement, number, figure or omission changed in a later version (the unit is declared in §5 and the count is recomputed by a script from the archived reports), the round produced 26 corrections across eight preprints plus one to the published estimator, and nine new versions were deposited. Its four structural findings were: criteria tested against noise but never against trivial predictors; verifiers that checked arithmetic but not inference; an estimator used for eight months without a known-truth bench; and a confounder the program had already sealed and never propagated. Round two (2 September) was a wider audit — 281 agent runs across thirteen lenses, two sceptics per finding, 130 findings surviving refutation — and produced 75 corrections: 64 across the ten preprints and 11 in the first version of this note, which had misdescribed its own evidence (it claimed passing verifiers for all ten preprints, blind reviewers, archived preprints and an error count of nine). Four of its findings we class as lethal: two mechanical verifiers written after round one had been emptied by a later edit and were approving with zero assertions; a reliability ceiling described in one abstract as measured "from order to chaos" came from ten networks all inside the chaotic band (+0.952 with the exact networks, not the +0.976 quoted) — and three reviewers of the same model lineage had accepted the statement without opening the script; none of the eleven public code packages ran outside the author's machine, while a check named "packages run from inside the zip" was green because it ran on that machine; and no paper stated the model's scope, the prior edge-of-chaos literature, or the fate of the program's predictions about human brains. We tabulate every sealed protocol by substrate: of 32 protocols with archived verdicts, 19 are in silico, 7 are within-brain mouse contrasts, and 6 are on human data, and no sealed prediction about human brains has been confirmed (exp21, exp55, exp56, exp56b). We report what neither round found: across 28 confirmatory result files, none predates the commit of its seal; among 18 sealed confirmatory runs, no two share a seed; the simulation engine reproduces its analytic ground truth to five decimal places. The program's method now carries eleven rules and six clauses on the adversarial step, each named with the failure that paid for it. The auditor was not external review and not peer review: it was an adversarially prompted instance of the same class of AI system that did the work, and round two showed the blind spot that arrangement cannot close on its own — a provenance statement that sounds methodological is accepted by reviewers of the same lineage unless one of them runs the code.
No takes yet. Share an insight, caveat, or question.
Nicolás Federico Galindez (2026) studied this question.