Reference quality of OpenEvidence across five medical specialties

Q: What does this research mean for the field?

The retrieval-augmented clinical AI tool OpenEvidence provides recent, high-impact, and non-fabricated references across multiple medical specialties, with only rare attribution errors. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

T.L. Lotan; Mahmud Omar; Yiftach Barash; Jonathan B. Kruskal; Muneeb Ahmed; Olga  R. Brook; Eyal Klang; Alon Gorenshtein

doi:10.1038/s44401-026-00142-8

September 2, 2026npj Health SystemsOpen Access

Reference quality of OpenEvidence across five medical specialties

View Full Paper

Ask AI

Bookmark

Share

Key Points

To evaluate the authenticity, recency, and bibliometric characteristics of references generated by the retrieval-augmented clinical AI tool OpenEvidence across five medical specialties.
Assessed 4,979 citations returned by OpenEvidence in response to 150 standardized clinical prompts across five medical specialties.
Evaluated citation existence, fabrication rates, abstract attribution errors, publication recency, and journal impact metrics.
Zero of the 4,979 evaluated references were confirmed as fabricated.
Three abstracts contained attribution errors, with the vast majority of references being recent and published in high-impact journals.
Journal concentration patterns varied across the five medical specialties.

Ask about this paper

Answers are grounded in this paper and cite their sources.

Implication

Evaluation study reveals zero fabricated citations across 4,979 references in clinical AI prompts, indicating reliable citation retrieval.

Abstract

OpenEvidence (OE) is a retrieval-augmented clinical AI tool. We evaluated the references OE returned to 150 standardized prompts across five specialties, analyzing all 4979 citations. No reference was confirmed as fabricated; three abstracts had attribution errors. Most references were recent and high-impact, though journal concentration varied by specialty. This study assessed reference existence and characteristics, not whether references support OE’s clinical claims - a benchmark for evaluating clinical AI tools.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Lotan et al. (2026) studied this question.

synapsesocial.com/papers/6a993653636c6408cfa7f5e9 https://doi.org/10.1038/s44401-026-00142-8

View Full Paper

Ask AI

Bookmark

Share

Reference quality of OpenEvidence across five medical specialties

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Reference quality of OpenEvidence across five medical specialties

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading