When the Scoreboard Lies

Retrieval-augmented generation depends heavily on preprocessing: how documents are cleaned, cut and enriched before they are indexed. The tools used to compare those choices are weaker than most teams assume. Across six pre-registered experiments (about $55 of model calls in total), we found that the standard scoreboard had credited our own preprocessing tool with an improvement that was really chunk size plus a subsidy built into the metric itself. We also found that a widely used AI-judge evaluation suite inflates scores for fluent text and quietly mishandles its hardest cases; that contextual retrieval, the field's favourite enrichment technique, genuinely works where it works but is corpus-dependent; and that the one place our tool did help sits in a layer existing metrics never isolate. This paper tells that story in order and describes the instrument that made each step measurable. The instrument is judge-free and deterministic: every text unit permanently carries the exact character ranges of the original documents it derives from, so scoring is interval arithmetic against registered answer spans — reproducible to the digit, and immune to the artifacts that misled us. The complete experimental record — apparatus code, pre-registrations frozen before results, all eleven experiment plans, every raw model reply, and every deviation ruled on in writing — is released alongside this paper at https://github.com/shamiksaharcciit-oss/provenance-rag-eval. Nothing in this paper asks to be believed; all of it asks to be re-run.