Key result
Multimodal LLMs show low ECG accuracy, scoring ~6 points higher on single- vs multi-pathology tracings.
Why the study?
Clinicians increasingly use multimodal large language models to interpret 12-lead ECG images without formal validation of their diagnostic accuracy. Existing benchmarks report overall accuracy without identifying which pathologies models interpret well or poorly, preventing targeted improvement and pathology-specific safety thresholds.
Does electrocardiogram pathology and complexity determine the diagnostic accuracy of multimodal large language models in interpreting 12-lead ECGs across 22 diagnostic categories?
Does electrocardiogram pathology and complexity determine the diagnostic accuracy of multimodal large language models in interpreting 12-lead ECGs across 22 diagnostic categories?
Multimodal large language models fail to achieve clinically acceptable diagnostic accuracy across any of 22 ECG pathology categories, demonstrating severe deficits in integrative diagnostic synthesis despite recognizing isolated waveform features.
Multimodal LLMs show limited ECG accuracy in complex cases; leaves open clinical utility pending prospective validation.
Background Clinicians increasingly use multimodal large language models to interpret 12-lead electrocardiograms (ECGs) from images, without formal validation of their diagnostic accuracy. Benchmark studies report overall accuracy but not which pathologies these models interpret well or poorly. This gap prevents targeted model improvement and makes it impossible to set pathology-specific safety thresholds for clinical use. Purpose We hypothesised that large language model accuracy for ECG interpretation degrades with increasing ECG complexity, particularly when multiple pathologies require integrative reasoning. We aimed to quantify performance across 22 diagnostic categories and identify which interpretive domains drive errors in each. Methods Three multimodal large language models (ChatGPT-5.1, Claude 4.5, Gemini 3) each interpreted 599 real-world 12-lead ECGs from a published five-volume teaching case series under minimalist and structured prompt conditions (3,594 reports total). We classified ECGs into 22 predominant pathology categories and scored each report using a prespecified, clinically weighted rubric (0-100) across ten interpretive domains. We fitted mixed-effects models with pathology, model identity, and prompt style as fixed effects, ECG case as a random intercept, and a domain-by-pathology interaction term to test whether specific domains varied in difficulty across diagnoses. Results Performance differed significantly across pathology categories (F=7.15) but was uniformly low (Figure 1). Normal ECGs scored highest yet reached only 63.6/100. Combined rhythm and conduction abnormalities scored lowest: narrow complex tachycardia with heart block 38.5 and atrial fibrillation/flutter with heart block 39.2 - a 25-point gap. Single-pathology ECGs outperformed multi-pathology ECGs (48.4 vs 42.1). A significant domain-by-pathology interaction (F=18.82) showed that errors were domain-specific: chamber hypertrophy recognition exceeded 70% across most pathologies, but final diagnostic synthesis fell below 5% in nearly all categories. Even for normal ECGs, only 20% of reports achieved a correct final diagnosis. Conclusion Large language model accuracy for ECG interpretation varies substantially by pathology, but no diagnostic category achieved clinically acceptable performance. Models reliably recognise isolated waveform features such as chamber hypertrophy but fail at integrative diagnostic reasoning across all pathology categories, with accuracy declining predictably as ECG complexity increases. Pathology-specific benchmarks are needed to identify which diagnostic categories require targeted model improvement and to define domain-specific safety thresholds before these tools are integrated into clinical ECG interpretation workflows.
No takes yet. Share an insight, caveat, or question.
Kamalanathan et al. (2026) studied 12-lead ECG interpretation across 22 predominant pathology categories (n=599). Multimodal large language models (ChatGPT-5.1, Claude 4.5, Gemini 3) interpreting 12-lead ECGs vs. 22 pathology categories and single- vs multi-pathology ECG complexity was evaluated on Diagnostic accuracy scored via a prespecified, clinically weighted rubric (0-100) across ten interpretive domains (F=7.15). Multimodal large language models showed uniformly low ECG diagnostic accuracy across 22 pathologies (F=7.15), scoring higher on single- vs multi-pathology ECGs (48.4 vs 42.1).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: