What question did this study set out to answer?

This study aims to evaluate the clinical reasoning capabilities of LLMs and RAG systems in managing pediatric myopia.

May 9, 2026Scientific Reports2 citationsOpen Access

Evaluation of large language models and retrieval-augmented generation for clinical reasoning in pediatric myopia: a 50-case real-world study

DKDaohuan KangChildren's Hospital of Zhejiang University KZKaikai ZhaoChina University of Mining and Technology DCDeji ChengHangzhou Medical College

Key Points

This study aims to evaluate the clinical reasoning capabilities of LLMs and RAG systems in managing pediatric myopia.
Real-world evaluation of six models: baseline LLMs and their RAG variants.
Performance assessed via automated scoring and blinded adjudication by three senior ophthalmologists.
Integration of 41 authoritative guidelines to enhance RAG systems.
GPT-RAG achieved a weighted automated score of 7.46, significantly higher than GPT Base's score of 7.37.
RAG models demonstrated 90–94% consensus alignment compared to 68–82% for baseline models.
High-risk recommendation probability was eliminated in all RAG models, while baseline models had error rates of 6–14%.

Abstract

To evaluate the clinical reasoning ability of large language models (LLMs) and retrieval-augmented generation (RAG) systems in pediatric myopia management using a real-world, expert-annotated case set covering diverse refractive, pathological, and high-risk scenarios. Six models were tested: baseline LLMs (GPT Base, Gemini Base, Grok Base) and their RAG variants (GPT-RAG, Gemini-RAG, Grok-RAG). RAG was augmented with 41 authoritative guidelines, including IMI white papers and the LAMP study. Performance was evaluated through automated scoring by Claude 4 Opus and blinded adjudication by three senior ophthalmologists, focusing on Accuracy, Utility, and Safety. RAG-enhanced models significantly outperformed baseline models across all metrics. Notably, GPT-RAG achieved the highest weighted automated score (7.46), surpassing GPT Base (7.37). Human adjudication revealed that RAG models achieved 90–94% consensus alignment compared to 68–82% for baselines. Crucially, the probability of high-risk recommendations—those capable of causing severe vision loss—was eliminated (0%) in all RAG models, whereas baseline models exhibited high-risk error rates of 6–14%. LLM + RAG integration boosts reliability and safety in pediatric myopia care, particularly for high-risk decisions. RAG’s domain knowledge incorporation advances AI clinical tools in ophthalmology, though ophthalmologist-in-the-loop refinement is essential pre-deployment.

Ask AI

Helpful

Bookmark

View Full Paper