Quantifying deep neural network uncertainty for atrial fibrillation detection with limited labels

Figures

• 

  Figure 1. A hypothetical example of how uncertainty estimates can help in interpreting ML model predictions. Here, an AF or sinus rhythm classifier is generating probability estimates for short segments of continuous ECG data. Adding an uncertainty estimate allows us to highlight the transition between different rhythm types, as shown in the raw (unprocessed) signal in red. Depending on the use case, these transition periods could be flagged for further inspection or ignored in favour of less ambiguous time intervals.

• 

  Figure 2. High-level description of the experimental pipeline and desired clinical application. The KGH cohort data were split into a small test set labelled by expert annotators and a much larger, unlabelled training set. Unlabelled ECG waveforms from this cohort dataset were fed into a surrogate model (pre-trained ECG classifier) to generate weak labels for the unlabelled training data. These were then used to train a new model from scratch with weak supervision techniques. Evaluating calibration and model uncertainty has a direct application for deployment (right), as it ensures the model can provide a reliable estimate of its confidence on previously unseen data.

• 

  Figure 3. Overview of weak supervision with confident learning. Network predictions are combined with weak labels to generate the confident joint, Which in turn is used to identify potentially incorrect labels for cleaning. The set of cleaned labels is used for further training.

• 

  Figure 4. Visual comparison of aleatoric and epistemic uncertainty estimation techniques. (a) Test-time augmentation randomly samples multiple views of each input. (b) Monte Carlo dropout samples multiple configurations of the model weights by randomly excluding a subset of neurons.

• 

  Figure 5. Example, unprocessed segments demonstrating different types of noise in the labelled KGH dataset. “No comment” indicates that while a record was identified to be noisy, there was no specific definition or terminology that clearly applied to the type of noise present. Only lead I is shown.

• 

  Figure 6. Calibration curves for all three models with various uncertainty estimation techniques applied, as compared to their fully certain baselines. (a) reliability curve of predicted probability (confidence) thresholds and the prevalence of the positive AF class from 0 (100% probability of sinus rhythm) to 1 (100% probability of the AF class). The dashed gray line represents a perfect calibration curve. (b) confidence versus number of samples with probability greater than the confidence threshold.

• 

  Figure 7. Density plots of uncertainty estimates using both MCDropout and test-time augmentation.

• 

  Figure 8. Comparison of model uncertainty versus confidence on in-distribution classes (AF, sinus rhythm) and out of distribution class (Abstain) when using both aleatoric and epistemic uncertainty estimation. All models show an association between high uncertainty and low confidence (i.e. neither definitively the AF nor the sinus rhythm classes). Compared to training with weak labels directly, Confident Learning avoids assigning low uncertainties to more non-confident samples that generally lean towards the AF class.