Answers are grounded in this paper and cite their sources.
Retrospective audit reveals discrepancies in LLM repair metrics across 2,400 runs, indicating that single success scalars fail to capture true benchmark performance.
Repository-level repair systems emit many logged events before a verifier pass. A rate over one event is easy to misread when the report omits the event's operational meaning, opportunity budget, support unit, or verifier boundary. We introduce a claim-indexed measurement model that binds each repair statistic to five coordinates: construct, opportunity design, unit of support, endpoint interpretation, and instrumentation provenance. We instantiate the model in a retrospective audit of 2,400 executions: two 30-task DeepSeek strata with disjoint task IDs and a 30-task GLM boundary stratum that reuses the clean task surface. A field-integrity audit identifies 12 raw GLM action positives that conflict with the patch-application evidence required by the governed-action construct. The harmonized table contains no verifier pass outside the governed-action state, while conditional endpoint yield is 14/199 (7.04%) and 14/204 (6.86%) in the two DeepSeek strata. Under eight retained opportunities per fixed cell, governed-action discovery reaches 40.83% and 46.67%; verifier-pass discovery reaches 4.17% and 1.67%. The 28 passing rows reduce to seven positive cells, three tasks, and two task families. Endpoint controls produce the expected result for all six gold runs, six no-op runs, and nine reference-derived mutants. These findings show that repair evidence cannot be reduced to one success scalar. Field semantics, opportunity, support, verifier scope, and instrumentation determine what a reported number measures and which comparisons it can sustain.
No takes yet. Share an insight, caveat, or question.
Xue et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: