Signal or noise? Evaluating commonly used attribution methods for explaining deep neural networks in electrocardiogram classification

Figures

• 

  Figure 1. Examples of the four experiments conducted. Each panel illustrates one of the four experiments designed to evaluate the clarity and soundness of attribution methods. (1) Inter-method similarity compares attribution maps generated by different attribution techniques applied to the same model and ECG input. (2) Self-consistency assesses the reproducibility of attribution maps across models trained with different random seeds but on the same data. (3) Progressive layer randomization evaluates how attribution maps change as layers of a trained model are incrementally randomized, testing dependence on model weights. (4) Input data perturbation quantifies the impact on model performance when top-ranked input features, according to each attribution method, are iteratively perturbed.

• 

  Figure 2. Examples of the four experiments conducted, each illustrating one example with a high score and one with a low score. Each coloured line represents a separate attribution plot, with the mean pair-wise correlation or predicted probability change displayed in the top right corner. The second experiment examples use Guided GradCAM as attribution method. The third experiment uses Guided GradCAM for the first example, and SmoothGrad for the second example. The fourth experiment displays Integrated Gradients for the first example, and SmoothGrad for the second example. Abbreviations: DS, DeepLiftSHAP; GS, GradientSHAP; IS, IG SmoothGrad; IG, Integrated Gradients; DL, DeepLift; SS, Shapley sampling; GG, Guided GradCAM; GC, GradCAM; GB, Guided Backpropagation; GI, Gradient*Input; SG, SmoothGrad; SA, Saliency; 0–9, seed number; R, randomized; T, unrandomized.

• 

  Figure 3. (A) Inter-method similarity: Pearson correlation between attribution maps produced using different attribution methods, for the same model (seed) and data. Results were aggregated over nine classes and ten seeds. (B) Self-consistency: Pearson correlations between attribution maps across seeds produced by the same attribution method, averaged over ten seeds and nine classes. Abbreviations: IG, integrated gradients.

• 

  Figure 4. (A) Dependence on model weights: +progressive randomization of a model trained to detect atrial fibrillation. The y-axis shows the Pearson correlation between the attribution maps produced using the non-randomized model and each randomization step. The x-axis shows the amount of randomized convolutional layers, starting at the output layer of the network. (B) Ability to identify features important to model prediction: iterative perturbation of 1% of input data based on the highest absolute attribution values and the resulting model performance. The original data points are replaced with a linear imputation of the neighbouring data points, multiplied with Gaussian noise.