The Judge Protocol: Constituting Evaluators That Are Themselves Evaluated by Outcomes

Evaluation of AI systems increasingly rests on judges that are themselves AI systems, and the results those judges produce are reported as measurements. Yet the judges are typically unconstituted (no recorded mandate delimits what they may decide), unmeasured (their error rates under a null are unknown), and unaccountable (their verdicts are never scored against what subsequently happened). This paper specifies a protocol that subjects evaluators to the discipline expected of any measured instrument. A judge exists only through a recorded, append-only, hash-anchored mandate; every verdict ("call") must name the observable outcome it implies and the earliest point at which that outcome could be observed. Independence between judge and judged is enforced structurally, by a mechanical lineage test over digest-anchored manifests, and evaluation material is sealed in fact by a commit–then–reveal construction fixed before the material is used. Each call is then recorded as a prediction in an append-only outcome ledger, anchored before its outcome can be known, and a judge's standing is computed mechanically — by a cumulative-sum statistic over realized outcomes, with fixed, pre-registered weight-loss steps — rather than asserted. The recursion of evaluation terminates at outcomes, which no judge grades. The meta-evaluation statistic's false-alarm behaviour is not assumed but measured exactly, on synthetic data, before adoption; we report those measurements, including the dominant effect of outcome persistence, and we state the protocol's open parameters and unresolved problems — among them the selective-label gap in outcome observability — as plainly as its mechanisms.