Protocol study reveals an executable preregistration framework in large language model agent evaluation, suggesting machine enforcement prevents protocol deviations.
Preregistration is normally a promise: authors declare hypotheses and analyses before data collection and readers trust that the declaration constrained the work. This paper registers a study whose preregistration is instead partially executable — encoded in the same machinery that will run the study, so that several classes of protocol deviation are refused by the instrument rather than merely discouraged by the document. The planned study is a paired comparison of a strong security baseline (C3) against the same baseline plus a bounded assurance plane (C5) for LLM agents, over exactly 800 unique paired cases (400 fault/attack, 400 benign; 1,600 condition executions). The assurance plane under test — released publicly as mobius-governance v0.7 — provides bounded context scanning and typed action mediation whose central invariant is that a scanner verdict of admit is never an action authorization. The execution contract enforces, mechanically: single-use run authorization under a globally reserved nonce; receiver-owned assembly from raw evidence to oracle result to locked analysis rows; a fixed-sequence gate (P1 fault-escape reduction, RD at or below -0.10 with a bootstrap 95% upper limit below 0; P2 benign non-inferiority tested only if P1 passes); exact two-sided McNemar tests with a locked sign convention checked by a swap fixture; a stratified paired bootstrap with exactly 5,000 replicates; and a four-endpoint Holm family. At publication time the study is HOLD_NOT_EXECUTABLE: no protected case exists and outcome access is zero. What has been verified is the instrument, not the hypothesis. We publish the design now so that any future outcome claim can point to a dated, DOI-bound freeze that provably predates the data. Release type: methods and preregistration paper; no outcome data exist at publication time. AI co-observer: Claude Fable 5 and Claude Opus 5 (Anthropic), receipt-audit and drafting working method; harness engineering iterations also used GPT-5 Codex (OpenAI); the registered author is the human author alone.
No takes yet. Share an insight, caveat, or question.
Toeda Taiko (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: