What does this research mean for the field?

canon-bench provides a rigorous protocol and benchmark to isolate and measure a coding agent's ability to carry over and respect constraints from previous sessions, preventing false passes caused by repository state or model priors. Novelty: ClaimNovelty.METHODOLOGICAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To evaluate whether autonomous coding agents reliably respect unmentioned constraints introduced in earlier sessions without relying on repository artifacts, model priors, or harness leakage.

September 5, 2026Open Access

canon-bench: Measuring Project Knowledge Across Agent Sessions

View Full Paper

Ask AI

Bookmark

Authors

SCShane Conner

Discussion

Member takes

Implication

Benchmark evaluation demonstrates rigorous protocols preventing false-positive memory carryover in autonomous coding agents, highlighting critical failure modes in cross-session persistence.

Key Points

To evaluate whether autonomous coding agents reliably respect unmentioned constraints introduced in earlier sessions without relying on repository artifacts, model priors, or harness leakage.
Constructed multi-session chains consisting of small repositories evaluated across four sequential sessions over a single work tree, designed such that obvious solutions fail external grading.
Implemented validator-backed checks with cold calibration under a frozen base worker to qualify tasks and screen predeclared facts prior to recall grading.
Conducted a demonstration campaign evaluating five chains under four experimental arms across five repetitions with cold controls.
Identified that artifact leakage into repository checkouts systematically causes false-positive constraint adherence during cross-session evaluations.
Formalized an invariant contract that maps and mitigates every unintended transmission channel between agent sessions.
Demonstrated qualitative boundaries of genuine knowledge carryover across five multi-session chains, delineating what persistent coding agent workflows reliably achieve.

Cite This Study

Shane Conner (2026) studied this question.

synapsesocial.com/papers/6a9bd49d6b95aff0620ec65e https://doi.org/10.5281/zenodo.22263116

View Full Paper

Ask AI

Bookmark

Implication

Benchmark evaluation demonstrates rigorous protocols preventing false-positive memory carryover in autonomous coding agents, highlighting critical failure modes in cross-session persistence.

canon-bench: Measuring Project Knowledge Across Agent Sessions

Discussion

Member takes

Implication

Key Points

Cite This Study

canon-bench: Measuring Project Knowledge Across Agent Sessions

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider