Multimodal large language models demonstrate consistently low accuracy for 12-lead electrocardiogram interpretation: a 599-case benchmark study

Background Multimodal large language models capable of image-based electrocardiogram (ECG) interpretation are increasingly used in clinical practice without formal validation. While task-specific ECG algorithms perform well for narrow classifications, the diagnostic reliability of general-purpose models across core ECG interpretation domains, and their sensitivity to prompt specification, remain unknown. No large-scale, domain-level benchmark has compared these systems for complete ECG interpretation. Purpose We aimed to benchmark the accuracy of three widely used multimodal large language models for complete 12-lead ECG interpretation using a clinically weighted scoring framework, and to determine whether structured prompt optimisation improves performance. Methods Three commercial multimodal large language models (ChatGPT-5.1, Claude 4.5, Gemini 3) each interpreted 599 real-world 12-lead ECGs drawn from a published five-volume ECG teaching case series, with textbook-assigned expert interpretations as the reference standard. Each ECG was interpreted under minimalist and structured prompt conditions (3,594 reports total). Four physician raters scored outputs using a prespecified, domain-weighted rubric (total 0-100) spanning fundamental parameters (rate, rhythm, axis, intervals) and higher-order interpretation (ischaemia/infarction, conduction abnormalities, chamber hypertrophy, final diagnosis). We analysed results using linear mixed-effects models with sensitivity analyses including unweighted scoring and bootstrap resampling (1,000 iterations). Results Overall performance was uniformly low (Figure 1). Under minimalist prompting, adjusted mean total scores (/100) were 48.7 for ChatGPT, 42.8 for Claude, and 43.6 for Gemini. Structured prompting yielded only marginal gains (50.3, 43.0, and 47.2, respectively); a significant improvement occurred only for Gemini (+3.58; P=0.0002). Domain-level analysis revealed errors concentrated in clinically critical domains: only 3% of responses achieved a perfect score for final ECG diagnosis and 32% for ischaemia/infarction, compared with 68% for chamber hypertrophy and 60% for QRS morphology. Bootstrap resampling confirmed stable model rankings across 1,000 iterations under both scoring schemes. Conclusions Widely used multimodal large language models demonstrated consistently low accuracy for complete ECG interpretation, with errors concentrated in clinically critical domains including ischaemia detection and final diagnosis. Structured prompt optimisation provided limited, model-specific gains without overcoming core interpretive limitations. General-purpose large language models are not suitable for standalone ECG interpretation in their current form.