Comparative evaluation of multimodal large language models for diagnostic accuracy in pediatric electrocardiography: a prospective comparative diagnostic accuracy study

Q: What is the clinical evidence from this study?

Study design: Cross-Sectional. Population: Pediatric ECG abnormalities (n=264). Intervention: Multimodal large language models (ChatGPT, Gemini 3, Microsoft Copilot) vs. Expert consensus (three pediatric cardiologists). Primary outcome: Positive likelihood ratio (+LR) for clinically significant ECG abnormalities (+LR 2.05, 95% CI 1.42-2.96).

Q: What does this research mean for the field?

Multimodal large language models demonstrate limited diagnostic utility for detecting clinically significant pediatric ECG abnormalities, with ChatGPT achieving the highest positive likelihood ratio of 2.05. Novelty: ClaimNovelty.NOVEL_FINDING. Consensus alignment: ConsensusAlignment.NEUTRAL.

Uğur Saraç; Ayşe Büşra Paydaş; Mustafa Gençeli; Talha Üstüntaş; Mehtap Yücel; Abdülkerim Çokbiçer; Fatih Şap; Tamer Baysal; Mehmet Burhan Oflaz

doi:10.1007/s00431-026-06874-x

March 24, 2026European Journal of PediatricsOpen Access

Comparative evaluation of multimodal large language models for diagnostic accuracy in pediatric electrocardiography: a prospective comparative diagnostic accuracy study

View Full Paper

Ask AI

Bookmark

Share

Key result

Multimodal large language models show limited diagnostic utility for detecting clinically significant pediatric ECG abnormalities.

+LR 2.05
95% CI 1.42-2.96
n=264

Figures

Why the study?

Head-to-head comparative diagnostic accuracy studies of multimodal large language models in pediatric ECG evaluation using likelihood ratios were lacking.

Does multimodal LLM interpretation improve diagnostic accuracy for clinically significant and emergency abnormalities in pediatric patients undergoing ECG evaluation?

Key Points

To evaluate and compare the diagnostic accuracy and clinical utility of three multimodal large language models (ChatGPT, Gemini, and Copilot) in interpreting pediatric electrocardiograms using likelihood ratios as primary outcomes.
Conducted a prospective comparative diagnostic accuracy study evaluating 264 pediatric patients (median age 108.0 months; 43.6% female) classified into Tier 1 (normal, n=120), Tier 2 (abnormal non-urgent, n=122), and Tier 3 (urgent emergency, n=22).
Evaluated multimodal LLMs—ChatGPT (GPT-5.2), Gemini 3, and Microsoft Copilot—against an expert reference standard.
Calculated sensitivity, specificity, area under the receiver operating characteristic curve (AUC), and positive (+LR) and negative (-LR) likelihood ratios.
For detecting clinically significant abnormalities (Tier 2 + 3 vs Tier 1), AUC values were 0.623 (95% CI = 0.555–0.690) for ChatGPT, 0.583 (95% CI = 0.513–0.652) for Gemini, and 0.550 (95% CI = 0.480–0.620) for Copilot.
ChatGPT showed the highest specificity (76.7%; 95% CI = 68.3–83.3%) with sensitivity of 47.9% (95% CI = 39.9–56.0%) and +LR of 2.05 (95% CI = 1.42–2.96), while Gemini had the highest sensitivity (79.9%; 95% CI = 72.6–85.6%) but low specificity (36.7%; 95% CI = 28.6–45.6%) and -LR of 0.55 (95% CI = 0.37–0.82).
In the emergency arrhythmia subgroup (n = 22), Gemini achieved 100% sensitivity (95% CI = 85.1–100.0%) and a -LR of 0.07 (95% CI = 0.00–1.12), but showed poor specificity of 30.2% (95% CI = 24.7–36.2%) and a +LR of 1.40 (95% CI = 1.27–1.55).

Study Design

Type

Cross-Sectional (n=264)

Blinding

Single-blind

Multicenter

No

Structured PICO

Does multimodal LLM interpretation improve diagnostic accuracy for clinically significant and emergency abnormalities in pediatric patients undergoing ECG evaluation?

P

Population

264 pediatric patients under 18 years of age with complete clinical assessments and technically adequate 12-lead ECG recordings

I

Intervention

Multimodal Large Language Models (ChatGPT GPT-5.2, Gemini 3, and Microsoft Copilot) interpreting 12-lead ECG images via a standardized zero-shot prompt

C

Comparator

Definitive reference standard diagnosis established by majority vote of three pediatric cardiologists

O

Outcome

Diagnostic accuracy using positive and negative likelihood ratios (+LR and -LR) for clinically significant abnormalities and emergency abnormalities

Main Result

Effect estimate: +LR 2.05 (95% CI 1.42-2.96)

Current multimodal LLMs show limited diagnostic utility in pediatric ECG interpretation and cannot be used as standalone diagnostic tools, though they may serve as adjunctive screening aids under clinician oversight.

Limitations

Conducted at a single tertiary clinic, limiting generalizability to primary care settings
ECG images were standardized using a flatbed scanner, which does not reflect real-world acquisition via smartphone photography
The emergency subgroup contained only 22 cases, precluding reliable subtype-specific estimates
Interobserver agreement among cardiologists was not formally quantified prior to majority adjudication
The zero-shot single-label prompt may have disadvantaged models in cases with multiple coexisting abnormalities
Formal quantification of interobserver agreement prior to majority adjudication was not performed
Small emergency subgroup size (n = 22) resulting in wide confidence intervals

Implication

LLMs should not be used standalone for pediatric ECG interpretation; leaves open adjunctive screening potential under clinician supervision.

Abstract

• This is the[FCA1] first head-to-head comparative diagnostic accuracy study of multimodal LLMs in pediatric ECG evaluation, using likelihood ratios as primary outcome measures. • All three LLMs showed limited rule-in utility (+LR near 1.0); Gemini achieved potentially meaningful rule-out performance for emergency arrhythmias (-LR = 0.07), but with wide confidence intervals reflecting the small emergency subgroup (n = 22). • Gemini's 100% sensitivity in the emergency subgroup reflects overcalling (specificity 30.2%) consistent with a triage/screening behavior rather than diagnostic precision.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Saraç et al. (2026) conducted a cross-sectional in Pediatric ECG abnormalities (n=264). Multimodal large language models (ChatGPT, Gemini 3, Microsoft Copilot) vs. Expert consensus (three pediatric cardiologists) was evaluated on Positive likelihood ratio (+LR) for clinically significant ECG abnormalities (+LR 2.05, 95% CI 1.42-2.96). Multimodal large language models demonstrated limited diagnostic utility for detecting clinically significant pediatric ECG abnormalities, with ChatGPT achieving the highest positive likelihood ratio of 2.05.

synapsesocial.com/papers/69c4ccf7fdc3bde448918b98 https://doi.org/10.1007/s00431-026-06874-x

View Full Paper

Ask AI

Bookmark

Share

Comparative evaluation of multimodal large language models for diagnostic accuracy in pediatric electrocardiography: a prospective comparative diagnostic accuracy study

Figures

Key Points

Study Design

Structured PICO

Main Result

Limitations

Implication

Abstract

Discussion

Cite This Study

Comparative evaluation of multimodal large language models for diagnostic accuracy in pediatric electrocardiography: a prospective comparative diagnostic accuracy study

Figures

Key Points

Study Design

Structured PICO

Main Result

Limitations

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading