Checks for Checkers: Sealed Paired Comparisons of AI Agent Verification Tools

We compared two AI agent verification tools in a sealed, blind, paired round of 96 planted and 96 clean cases. Tool B, an open-source evaluation framework, detected 88 planted cases. Tool A, the product of the author's startup, detected 83. This nominal gap is not a lead: the paired difference is -5.2 points (95% interval -14.1 to +3.0), and the exact McNemar p is 0.30. A second judge agreed on flag labels (Cohen's kappa 0.94) and found no lead. Neither tool raised a false alarm as defined, though Tool A flagged the same two pre-existing behaviours on all 48 clean builds of one application. Read literally, one sealed scoring step would have given a spurious 30-to-0 lead for Tool A. In an earlier round, one clarifying sentence in the original judge's instruction moved detection from 5 and 4 of 18 cases to 15 and 15. We rewrote operators' flags to make detected defects read as missed. Non-adaptive rewrites written to recast a defect's cause or present it as unmodified behaviour flipped up to 8 of 166 detections (4.8%) across five judges. Rewrites written to delete the specifics flipped 168 of 170 detections for the primary judge. We read that as the instruction working as written, not an attack: it asks the judge to match those specifics. Before either tool runs, the protocol, answer key and manifest are timestamped by a public authority, and verdicts are committed before the key opens. The cases and tool identities are withheld, and every operator came from one model family.