Answers are grounded in this paper and cite their sources.
Benchmark evaluation demonstrates rigorous protocols preventing false-positive memory carryover in autonomous coding agents, highlighting critical failure modes in cross-session persistence.
A coding agent can pass a later task without having remembered anything. The repository can carry the answer, the model's priors can supply it, or the harness can put it beside the work tree. canon-bench asks something narrower than memory: does an agent respect a constraint that an earlier session presented and that the later task never mentions? The first cut of one chain answered yes and was wrong: the planting session's own repair left the convention in the checkout, the probe read that file and copied it, and every check passed. The grade described the checkout. This paper describes the protocol built around that failure. A chain is a small repository and four sessions over one work tree, where the probe's obvious solution is wrong in a way only a grader outside the checkout can see. Candidate checks are authored and validator-backed, and a cold calibration under a frozen base worker decides which are eligible for a campaign. Predeclared plant-derived facts pass a cold screen before a judge grades recall. The paper maps every path by which the instrument can report carryover against what the protocol does about each, organizes the chain contract's invariants by the false pass each one closes, and reports one frozen demonstration campaign, five chains under four arms and five repetitions, stating what it establishes and what it cannot. The record holds the paper, its LaTeX and figure sources with the d3 figure pipeline, the claims contract that bounds every factual statement, the derived caption numbers, the outside reviews and their dispositions, the campaign results and cold controls, and the instrument's own documents. The instrument is open source at github.com/shaneconner/canon-bench under the MIT license; the paper and author-created artifacts are CC BY 4.0. This is a preprint and has not been peer reviewed.
No takes yet. Share an insight, caveat, or question.
Shane Conner (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: