Reliability of snapshot AI-ECG predictions: test-retest reproducibility across seven prediction tasks in three hospital cohorts

Background Artificial intelligence applied to the electrocardiogram (AI-ECG) enables scalable screening from a single recording. However, clinical deployment requires confidence that predictions remain stable across repeat ECGs within the same patient. The test-retest reliability of AI-ECG models across multiple clinical tasks has not been systematically evaluated yet. Purpose To quantify the short-term test-retest reliability of AI-ECG predictions across demographic, prognostic, and disease classification tasks in three independent hospital cohorts. Methods We trained residual convolutional neural network (ResNet) models on 94,770 patients from the Beth Israel Deaconess Medical Center dataset (BIDMC) database, for seven prediction tasks: two demographic (age and sex), one prognostic (5-year mortality), and four disease classifications: systolic heart failure (HF), atrial fibrillation (AF), valvular heart disease (VHD), and hypertension (HTN). The model was tested across three independent hospital cohorts: hold-out test split of BIDMC, Emory University Hospital (EUH), and Massachusetts General Hospital (MGH). Only clusters with more than five sequential ECGs recorded within 5-day windows were identified. Within-cluster reliability was quantified using intraclass correlation coefficients (ICC), standard error of measurement (SEM) and minimal detectable change (MDC), and for binary classifications, Cohen’s kappa (κ) and discordance rates. Results A total of 204,666 clusters were analysed (8,662 BIDMC; 12,307 in EUH; 183,696 in MGH). Across all tasks and cohorts, ICCs ranged from 0.66 to 0.84, indicating good-to-excellent reliability (Table 1). Demographic predictions were most stable, age demonstrated ICC 0.77–0.84 (SEM 5.0–6.0 years), while sex ICC was 0.67–0.78. Among disease and prognostic tasks, HF (ICC 0.79–0.81) and AF (ICC 0.77–0.82) showed the highest reliability, followed by VHD (ICC 0.71–0.76), HTN (ICC 0.68–0.76), and 5-year mortality (ICC 0.66–0.75). Cohen's kappa was also highest for HF and AF (κ 0.64–0.70; discordance rate 9–17%), followed by HTN (κ 0.54–0.62; discordance rate 19–23%) and 5-year mortality (κ 0.49–0.61; discordance rate 15–22%). Reliability patterns were consistent across external validation cohorts, with modest attenuation compared with the training cohort. Conclusion AI-ECG predictions demonstrate reproducible short-term test-retest reliability across seven clinical tasks and three independent hospital cohorts, supporting the use of snapshot AI-ECG predictions for cardiovascular screening. Conditions with distinct electrocardiographic signatures such as AF and HF yield the most stable predictions amongst disease tasks. These findings provide a reproducibility benchmark for AI-ECG disease classification and define measurement noise thresholds that can inform clinical interpretation of AI-derived cardiovascular risk scores.Graphical figure