Uncertainty estimation for deep learning-based automated analysis of 12-lead electrocardiograms

Figures

• 

  Figure 1. Overview of the uncertainty estimation concept and the epistemic uncertainty estimation methods. All methods work similarly to human uncertainty (in the top box, illustrated as several brains), where there are multiple reviewers interpreting the same electrocardiogram. The uncertainty is then calculated as the variance over these different predictions for the same electrocardiogram. With deep neural networks multiple predictions can be achieved using ensembles (i.e. training the same network multiple times), MC dropout (i.e. removing some nodes randomly during prediction), or variational inference (i.e. sampling from the same network with distributions as weights multiple times).

• 

  Figure 2. Flowchart of the possible clinical implementation. The deep neural network provides a certainty value first, which is compared with a predefined threshold. Only interpretations where the deep neural network is certain about are then automatically interpreted. The uncertain electrocardiograms are interpreted by the physician. This example shows an incorrect prediction by algorithm, where subtle ST-depressions are interpreted as aspecific ST-segment abnormalities. Due to the low certainty, this example will be passed on the physician for interpretation, that classified it as acute.

• 

  Figure 3. Calibration of the base network in the in-distribution (A) and out-of-distribution (B) setting for all datasets. In a calibration plot, the predicted probability or confidence of the network is grouped into ten bins from low (i.e. 20–30%) to high (i.e. 90–100%). For all these bins, the accuracy in that bins is calculated. A perfectly calibrated model outputs confidences that match up exactly with the accuracy. A model which predicts higher probabilities than the accuracy is over-confident, which can be observed by a line that falls under the diagonal. An under-confident model is the opposite and lies above the diagonal. The base models without uncertainty estimation are op to 30% over- or under-confident.

• 

  Figure 4. Clinical simulation with accuracies of predictions as a function of excluding uncertain electrocardiograms on the in-distribution setting of the UMCU-Triage dataset. The threshold percentage corresponds to the percentage of data that needs to be evaluated by a physician after exclusion. Through excluding uncertain electrocardiograms, accuracy of all models improved. The VI+BD model had the steepest upward slope, and thus excluded the uncertain electrocardiograms the fastest, thereby increasing overall model accuracy at the highest rate. AUX, auxiliary output; BD, Bayesian decomposition; ENS, ensemble; MCD, Monte-Carlo dropout; SSE, snapshot ensemble; VI, variational inference.

• 

  Figure 5. Normalized per-class thresholding plots of the VI+BD (A, B) and ENS+AUX (C, D) models on the UMCU-Diagnose dataset. The first column (A, C) is for the in-distribution setting, the second column (B, D) is for the out-of-distribution setting. Classes with high uncertainty are removed first and have a steep downward slope. In the in-distribution plots (A, C), the model was trained on all classes, including atrial fibrillation. These plots show that the algorithm is certain about prediction atrial fibrillation, as these samples are excluded slower than other classes. In the out-of-distribution plots (B, D), the algorithm was trained on a dataset that contained no atrial fibrillation electrocardiograms. These plots show that the model is now very uncertainty about this unseen class, as it excludes the atrial fibrillation electrocardiograms first. AF, atrial fibrillation; I-ABV, first degree atrioventricular block; LBBB, left bundle branch block; PAC, premature atrial contraction; PVC, premature atrial contraction; RBBB, right bundle branch block; STD, ST-depression; STE, ST-elevation.

• 

  Figure 6. Correspondence of uncertainty with cardiologist’s lack of agreement. The electrocardiograms in the expert test set of UMCU-Triage are grouped by consensus between two cardiologists and compared with the estimated uncertainty for these electrocardiograms, both per-class and overall. The algorithm is more certain about electrocardiograms where the cardiologists agreed. Moreover, the algorithm is most certain about the normal electrocardiograms and least certain about the abnormal, acute electrocardiograms, which is also the smallest class. **P < 0.01, **** P < 0.0001. ns, not significant.

• 

•