Key result
Vision-enabled LLMs show moderate ECG accuracy but low sensitivity and poor agreement with expert consensus.
Why the study?
Vision-enabled large language models show potential for medical image interpretation, but their real-world performance on 12-lead ECGs has not been systematically assessed.
Do vision-enabled large language models accurately interpret real-world ECG images compared to expert consensus?
Cross-Sectional (n=70)
Do vision-enabled large language models accurately interpret real-world ECG images compared to expert consensus?
Current vision-enabled large language models demonstrate insufficient diagnostic accuracy, low sensitivity for critical findings, and limited agreement with experts, making them unready for clinical deployment in ECG interpretation.
Answers are grounded in this paper and cite their sources.
VE-LLMs lack sufficient accuracy and agreement for clinical ECG use; cross-sectional data leaves open whether refined models can achieve expert reliability.
Background: Vision-enabled large language models (VE-LLMs) have the potential to provide flexible and explainable medical image interpretation. However, their real-world performance on clinical data such as 12-lead electrocardiograms (ECGs) has not been systematically assessed. Objective: This study aimed to evaluate the diagnostic accuracy and reliability of state-of-the-art VE-LLMs in interpreting real-world ECG images. Methods: We tested eight VE-LLMs (ChatGPT-5, ChatGPT-4, Gemini 2.5 Pro, Copilot, Grok-4, Perplexity, Claude Sonnet-4, and Claude Opus-4.1) using 70 de-identified ECG images. A standardized prompt requested nine determinations: rhythm, first-degree atrioventricular (AV) block, intraventricular conduction block and pattern, corrected QT (QTc) prolongation, premature atrial and ventricular contractions, ischemic ST-segment deviation, and axis deviation. An expert consensus served as the reference standard. Model outputs were evaluated using overall and per-class diagnostic metrics. Results: Overall accuracy across models varied significantly from 68.1% to 78.3% (429/630 to 493/630, Cochran’s Q, P<.001). ChatGPT-5 achieved the highest accuracy (78.3%) but had the slowest response time (median 276 seconds), whereas Perplexity and Copilot responded within a median of 2 and 3 seconds, respectively. Rhythm classification reached 72.9%–82.9% accuracy (51/70 to 58/70), but sensitivity for atrial fibrillation was ≤22% (≤2/9). Detection of first-degree AV block was poor (sensitivity 0%–33%; 0/9 to 3/9), and QTc prolongation was also poor (sensitivity 0%–45.5%; 0/22 to 10/22). Intraventricular block was identified with up to 70% accuracy (49/70), but correct subtype assignment was ≤44% (≤11/25). ST-segment deviation sensitivity was <25% for all models (highest 3/14). Agreement with expert interpretation was low, with Cohen’s kappa (κ) indicating poor-to-fair concordance (κ≤.37). Conclusions: VE-LLMs showed moderate overall accuracy but low sensitivity and limited agreement with expert ECG interpretation. Current performance is inconsistent and insufficient for clinical deployment. Future development should focus on domain-specific training and hybrid approaches combining LLM reasoning with established ECG algorithms before use in patient care.
No takes yet. Share an insight, caveat, or question.
Soubh et al. (2025) conducted a cross-sectional in Electrocardiogram abnormalities (n=70). Vision-enabled large language models (VE-LLMs) vs. Expert consensus was evaluated on Overall diagnostic accuracy (p=<.001). Vision-enabled large language models showed moderate overall accuracy for ECG interpretation (68.1% to 78.3%) but had low sensitivity and poor agreement with expert consensus (kappa ≤0.37).
Synapse has enriched 10 related papers. Consider them for comparative context: