What question did this study set out to answer?

This research aims to assess the accuracy and quality of responses generated by large language models in surgical patient education.

May 13, 2026Open Access

Generative AI in Surgical Care: Evaluating Large Language Model Performance in Patient Education

Key Points

This research aims to assess the accuracy and quality of responses generated by large language models in surgical patient education.
Conducted a systematic review per PRISMA guidelines
Included studies that presented patient-centered surgical questions to large language models
Used quantitative evaluation scales to assess LLM responses
ChatGPT scored 74.7% accuracy compared to 65.9% for other LLMs (p = 0.004).
LLMs scored significantly lower in quality than standard patient education materials (ChatGPT: 51.8 vs 65.6, p < 0.0001).
ChatGPT readability was higher than other LLMs (13.4 vs 10.5, p < 0.0001).

Abstract

Background: The advancement of generative artificial intelligence (AI) technologies is affecting how individuals seek and consume information. Large language models (LLMs) are increasingly being used for health-related inquiries, including surgical information, yet concerns remain regarding the quality of evidence-based content they provide. Methods: A systematic review per PRISMA guidelines was conducted to evaluate the accuracy, quality, and readability of LLMs in responding to surgery-related questions. Studies were retrieved from MEDLINE, CENTRAL, and EMBASE databases related to LLM-generated responses to patient-centered surgical inquiries (December 2024) and were included if they presented patient-centered surgical questions to an LLM, defined relevant outcome measures, and used quantitative evaluation scales to assess LLM-responses. Quantitative synthesis involved a linear mixed-effects model, using estimated marginal means and Holm-adjusted post-hoc comparisons to assess model performance. Results: 64 studies comprising 1,664 surgical questions across 12 surgical subspecialties were included. The most evaluated LLMs were ChatGPT-3.5 (70.3%) followed by ChatGPT-4.0 (37.5%) and Google Gemini (23.4%). ChatGPT (pooled analysis of versions 3.0, 3.5 and 4.0) scored higher in accuracy compared to other LLMs (74.7% ± 2.5 vs 65.9% ± 3.2, p = 0.004). However, all LLMs underperformed in quality compared to standard patient education materials, as measured by the DISCERN instrument. ChatGPT scored significantly lower (51.8 ± 1.8 vs 65.6 ± 2.4, p < 0.0001), as did other LLMs (49.7 ± 2.7 vs 65.6 ± 2.4, p < 0.0001), compared to standard patient education materials. Readability, measured by the Flesch-Kincaid Grade Level was higher in ChatGPT compared to other LLMs (13.4 ± 0.3 vs 10.5 ± 0.5, p < 0.0001). Conclusions: LLMs are an emerging tool in surgical patient education, but demonstrate limitations related to content quality. Further refinements in AI-generated health content can further improve clarity, reliability, and surgery-specific accuracy.

Ask AI

Helpful

Bookmark

View Full Paper