OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an Advantage?

PURPOSE: To assess the accuracy of OpenEvidence relative to six leading large language models (LLMs) in answering refractive surgery multiple-choice questions, and to compare their performance with users of the American Academy of Ophthalmology (AAO) Basic and Clinical Science Course (BCSC) Self-Assessment Program. SETTING: Not applicable. DESIGN: Randomized questionnaire-based study using 225 questions from the refractive surgery section of the AAO BCSC Program. METHODS: OpenEvidence and 6 LLMs (ChatGPT-4o, ChatGPT-5, Claude, Copilot, Gemini, and Grok) answered the same set of multiple-choice questions. Their accuracy was defined as the proportion of correct responses. Performance from 1,983 BCSC Self-Assessment Program users served as a benchmark. RESULTS: OpenEvidence achieved the lowest accuracy (76%) and performed significantly worse (P<0.05) than all six general-purpose LLMs. Claude achieved the highest accuracy (92%), followed by Grok (89.8%). ChatGPT-5, Copilot, and Gemini each scored an accuracy of 88%, and ChatGPT-4o achieved 86.2%. All models' lower 95% confidence interval bounds exceeded the 60% passing threshold. The average score of BCSC Self-Assessment Program users was 77%. All general-purpose LLMs scored significantly higher than BCSC Self-Assessment Program users (P<0.05), whereas OpenEvidence did not. CONCLUSIONS: OpenEvidence demonstrated significantly lower accuracy than the six evaluated general-purpose LLMs and did not exceed the BCSC Self-Assessment program user benchmark on this standardized refractive surgery multiple-choice assessment. This task-specific difference was not fully explained by response-format noncompliance and should not be interpreted as direct evidence of globally inferior medical knowledge or performance in open-ended clinical applications.