Answers are grounded in this paper and cite their sources.
Can a small, transparent, standard-library harness measure retrieval-augmented generation (RAG) quality honestly, and does a targeted fix for one failure class hold up outside the data it was built on? This paper reports an exploratory study, and says so in its title: the project was built without a locked plan, and its three hypotheses were written on 2026-11-07, after every result below already existed. They are scored post hoc and carry no confirmatory weight. The harness is a BM25 retriever over sentences with an extractive answer step, three lexical metrics with fixed pass lines, a budgeted judge layer, and a CPU embedding judge, run on synthetic sets of 8, 20, and 32 cases and on deterministic prefix samples of SQuAD validation (500 cases) and HotpotQA distractor (200 cases). H1 (the stemmed-overlap reranker raises the synthetic Deep pass rate) is supported in-distribution only: the Deep set went from 23 to 32 of 32 and the target failure class from 5 to 13 of 13, but 12 of the 13 cases in that class were written after the failure was seen. H2 (the reranker raises gold-in-top-passage on real data) is not supported: on SQuAD the gold answer reached the top passage in 315 of 500 cases without the reranker (Wilson 95 percent interval 0.587 to 0.671) and 319 with it (0.595 to 0.679), 26 helped against 22 hurt, exact sign test p 0.6655; on HotpotQA, 39 to 47 of 200, 14 helped against 6 hurt, p 0.1153. None of the 12 rerank tests survives a Bonferroni correction. H3 (span answers beat sentence answers on lexical pass) is supported, and it is the study's largest measured effect: SQuAD lexical passes rose from 24 to 113 of 500 without rerank (exact McNemar 21 against 110, p below 0.0001) and HotpotQA from 1 to 7 of 200 (0 against 6, p 0.0312). The embedding judge agrees with a gold-derived proxy label at Cohen's kappa 0.5607 for SQuAD sentence answers and 0.7111 for span answers at the shipped thresholds; the judge has not been validated against human labels, and no human labels exist in this study. Every scored result regenerates byte-identically from the committed evidence by one command. The contribution is a negative transfer result, published where it fell, and a small harness whose numbers a reader can check by hand.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: