Seeded-Error Testing: A Protocol for Assuring Human Oversight of AI

Organizations deploying AI in high-risk settings are increasingly expected to show that human oversight works. A reviewer who nominally checks outputs but accepts nearly everything provides little protection, and neither policies nor decision outcomes reveal this. Seeded-Error Testing proposes a protocol for measuring directly whether reviewers detect and correct AI errors. Known errors are inserted into the stream of cases that reviewers see, and their responses are scored against ground truth. The protocol has six steps: define the decision point, reviewers, and risk tier; design the error classes, prevalence, and case library; run the test with appropriate insertion mode, safeguards, and blinding; measure detection, false alarms, sensitivity, and response criterion; interpret results against thresholds set by risk tier; and report the findings with their limits and the triggers for re-testing. Grounded in signal detection theory, it separates how well reviewers discriminate errors (d′) from how readily they accept or challenge outputs (criterion c), because two teams with similar catch rates can differ greatly in sensitivity. It also addresses the number of seeded errors needed for usable precision, ethical and operational safeguards, and common design mistakes. It is intended for assessors, auditors, and standards bodies who need a defensible way to produce the highest level of evidence that oversight is effective.