Unrecorded Evaluation Provenance in a Clinical Diagnosis Benchmark: A Forensic Case Study of Scored-Artifact Mismatch

We report a single, forensically reconstructed incident in which a fixed-case benchmark used to tune a clinical differential-diagnosis system's ranking engine measured the output of an internal, partially-tuned candidate list rather than the list ultimately shown to a clinician, and no field in the system's run-tracking tables recorded which of the two a given benchmark run had actually scored. Reported top-1 accuracy on a 120-case internal suite rose from 35.0% to 55.8% over seven runs during a single tuning session; an independent remeasurement against the deployed decision path approximately ten days later returned 27.5% (displayed as 28%), which at face value suggests the benchmark had been badly inflated. We show that comparison is not supportable as stated: the 27.5% figure mixed two different scoring-engine configurations because of an unrelated, unpinned engine-selection mechanism; isolating each configuration separately gives 25.8% for the older configuration and 37.5% for the newer one, neither of which is a clean substitute for a measurement taken at the time of the session. We reconstruct the incident using version-control history and live application-database queries. Because our first-pass reconstruction itself converged on an overstated causal story under leading investigative prompts, we also conducted a second, deliberately blind and adversarial re-verification pass that narrowed the claim to what the evidence actually supports. We identify three concurrent contributors to the apparent benchmark-to-deployment gap, directly quantify one of them (an engine-selection confound in the available comparator), and show why the available evidence cannot separately estimate the other two's individual contributions. We argue this incident is an instance of a broader class of evaluation-harness failure: evaluation code that silently scores an unintended artifact, with no logged record of which artifact was scored. This particular instance was never reported anywhere and so is invisible to the public-issue-mining methodology used by the largest empirical studies of evaluation-harness defects we are aware of; we treat single-incident forensic reconstructions such as this one as a necessary complement to that methodology, not as evidence about the class's overall prevalence. We discuss implications for evaluation-harness design during rapid, AI-assisted iterative development, and for provenance requirements in clinical decision-support evaluation specifically, where downstream regulatory frameworks already condition non-device status on a system's outputs being independently reviewable.