What question did this study set out to answer?

This study aims to evaluate the diagnostic outputs of large language models compared to emergency physicians in critically ill patients in the emergency department.

July 17, 2026Open Access

Diagnostic capability of large language models in critically ill patients: a prospective single-centre study comparing ChatGPT, Claude, and Gemini with emergency physicians

İGİbrahim GünaydınKaradeniz Technical University MYMutlu YılmazKaradeniz Technical University SASinan AkpunarKaradeniz Technical University

Key Points

This study aims to evaluate the diagnostic outputs of large language models compared to emergency physicians in critically ill patients in the emergency department.
Prospective, single-centre observational study comparing outputs of four large language models with emergency physicians.
Participants included critically ill patients from August 10 to September 10, 2025, with clinical data analyzed using standardized prompts.
Cohen's κ was used to quantify diagnostic agreement between LLM and physician outputs.
Physicians' preliminary diagnoses matched the reference in 83.9% of cases, significantly better than LLMs (p < 0.001).
Final diagnosis match rates for LLMs ranged from 59.4% to 67.2%, with no significant pairwise differences (p = 0.16).
Cohen's κ values indicated moderate-to-substantial agreement, with LLMs consistently underperforming compared to physicians.

Abstract

BACKGROUND: Clinical decision-making requires integrating history, physical examination, laboratory, and imaging data. In the emergency department (ED), workload, time pressure, and cognitive burden may impair this process and affect decision quality. This study compares the diagnostic outputs of ChatGPT, Claude, and Gemini with those of emergency physicians in real-world ED cases. METHODS: This prospective, single-centre observational diagnostic agreement study compared the stage-wise outputs of four Large Language Models (LLMs) (ChatGPT-4o, ChatGPT-5, Claude Opus 4.1, and Gemini 2.5 Pro) with those of emergency physicians in critically ill ED patients. Between 10 August and 10 September 2025, de-identified clinical data were entered into the models via their official web interfaces using standardised prompts. In the first stage, physicians and LLMs each generated five preliminary diagnoses based on vital signs and medical history. In the second stage, following physical examination and laboratory and imaging results, both refined their lists into three differential diagnoses. In the third stage, the physicians' final diagnosis was accepted as the reference, and each LLM was prompted to provide a final diagnosis. LLM preliminary and differential diagnoses were compared with those of the physicians at the corresponding stage, and LLM final diagnoses with the reference; the inclusion of the final diagnosis within earlier lists was also evaluated. Agreement was quantified using Cohen's κ; analyses were performed in R. RESULTS: Of 389 screened patients, 180 were included (56.1% male; mean age 67 ± 15.9 years). Physicians contained the reference diagnosis within their top-5 preliminary and top-3 differential lists in 83.9% and 98.3% of cases, respectively, significantly exceeding every LLM (all p < 0.001). Final-diagnosis match rates were 67.2% 60.3-73.5 for ChatGPT-4o, 65.6% 58.7-71.9 for ChatGPT-5, 63.3% 56.3-69.9 for Claude Opus 4.1, and 59.4% 52.3-66.1 for Gemini 2.5 Pro (p = 0.16). Cohen's κ ranged from 0.575 (Gemini 2.5 Pro) to 0.656 (ChatGPT-4o), indicating moderate-to-substantial agreement, with no pairwise difference reaching significance. CONCLUSIONS: The LLMs achieved moderate agreement with ED reference diagnoses in critically ill patients but were consistently outperformed by physicians at the early diagnostic phases. Despite final-diagnosis match rates of 59%-67%, their current diagnostic role in the ED remains limited.

Ask AI

Helpful

Bookmark

View Full Paper