Answers are grounded in this paper and cite their sources.
Background We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged. Objective We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates. Methods Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations. Results Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis. Conclusions Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.
No takes yet. Share an insight, caveat, or question.
Hasei et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: