Towards artificial intelligence-based learning health system for population-level mortality prediction using electrocardiograms

Figures

• 

  Fig. 1. Study cohort, data splits and evaluation scheme. Flowchart of study design, showing sample sizes for overall study, experimental splits and different outcomes. We divided the overall ECG dataset into a random split of 60% for the model development (for training (90% subset of the 60%) and fine tuning (10% subset of the 60%) the model), and the remaining 40% as the holdout set for validation. Patients with more severe illnesses are expected to undergo ECGs more frequently, and more ECGs may be acquired at terminal stages when patients are monitored more regularly. To mitigate potential bias in model performance due to differential representation of patient phenotypes, we evaluated our models using a single randomly-selected ECG per patient from their multiple episodes in the holdout set.

• 

  Fig. 2. Comparison of model performances. Comparison of AUROC model performances for DL and XGB models with ECG traces and measurements. Error bars show 95% confidence intervals around the mean. DL with ECG traces, age and sex was the best model in this comparison, with AUROCs consistently higher than 0.82. AUROC Area under the receiver operating characteristic curve; DL deep learning; ECG electrocardiogram; XGB XGBoost.

• 

  Fig. 3. Predicted risk groups for 1-year mortality. a We derived five risk groups - ‘very low’, ‘low’, ‘medium’, ‘high’, ‘very high’ risk groups based on 20 percent cut-points (0 - 20%, 20% - 40%, etc.) of predicted probability of death from our main models (DL: ECG Trace, age, sex) in the holdout set. b Percentage of observed deaths in each predicted risk group showed good calibration with a steady increase across the risk groups (8.6%, 34.6%, 52.3%, 70.9%, and 78.9% death in the ‘very low’, ‘low’, ‘medium’, ‘high, and ‘very high’ risk groups, respectively). DL deep learning, ECG electrocardiogram.

• 

  Fig. 4. Model performances in diagnostic and sex-based subpopulations. a Performance of DL: ECG traces, Age, Sex models in different primary diagnosis subgroups. The models performed better in patients with STEMI and NSTEMI (AUROC of 0.867 and 0.882 for 1-year mortality, respectively) than in the overall cohort. The performance of the model in the other subgroups (heart failure, diabetes and atrial fibrillation) was lower than in the overall holdout cohort. b The prognostic models performed slightly better in men than in women. AUROC Area under the receiver operating characteristic curve, DL deep learning, ECG electrocardiogram, NSTEMI non-ST elevation myocardial infarction, STEMI ST elevation myocardial infarction.

• 

  Fig. 5. Comparison of model performances with and without lab features. Error bars show 95% confidence intervals around the mean. ECG models had higher AUROCs than the baseline model (XGB: Age, Sex, Lab) for all time points, even without lab values, but the difference was smaller for longer range predictions. Addition of lab features significantly improved the model performances throughout, across models and time points, however the gains in performance were small in magnitude (0.99% on average, DeLong Test, all p < 0.001). Overall, the DL model with ECG traces, age, sex and lab was the best performing model in the comparison. AUROC Area under the receiver operating characteristic curve, DL deep learning, ECG electrocardiogram, XGB XGBoost.

• 

  Fig. 6. GradCAM heatmap for 1-year mortality. Representative ECG traces were chosen with primary diagnoses of AF, HF, STEMI, and NSTEMI. The darker areas in each trace on GradCAM denote the areas with the most contribution to ResNet DL: ECG model’s 1-year mortality prediction. The highlighted regions are not lead-specific, and are driven based on data from all 12 leads. AF atrial fibrillation, DL deep learning, ECG electrocardiogram, HF heart failure, STEMI ST-elevation myocardial infarction, NSTEMI Non-ST-elevation myocardial infarction.

• 

  Fig. 7. SHAP based feature importance for 1-year mortality. a SHAP analysis of the XGBoost models with ECG measurements, age, sex showed that higher age, lower RR interval, horizontal QRS axis (conditional effect), higher Bazett’s rate-corrected QT interval, male sex and lower PR interval contributed the most to the 1-year mortality. b The addition of lab features highlighted contributions of lower hemoglobin, lower glomerular filtration rate (GFR), lower troponin I, higher creatinine, very high or low sodium and high potassium. The description of ECG measurements is provided in the Supplementary Table 5.

• 

  Fig. 8. Schematic of prediction tasks, feature types and choice of learning algorithms. a This study focused on developing and evaluating ECG based mortality models to predict the probability of a patient dying within 30-days, 1-year and 5-years, starting from the day of ECG acquisition. ECGs used in these models could have been acquired at any time point during a healthcare episode. Models included features with i ECG only, ii ECG + age, sex, and iii ECG + age, sex + lab tests. The goal of the prediction models is to output a calibrated probability of mortality, which could be used in patient risk-assessment. b Patient’s ECG data are generally archived by healthcare facilities as one of two formats: either i as a clinical report of summarised ECG measurements such as QT interval, QRS duration etc. or ii less commonly, as raw voltage time series of ECG signal tracings. In order to facilitate wider applicability, we used learning algorithms that are appropriate for the data formats, namely ResNet based deep learning for the information-rich multi-channel voltage time series and gradient boosting-based XGBoost for the ECG measurements.