Decoupling Visual Saliency from Decision Logic: A Fidelity Evaluation of DAAM in Vision Transformers

Vision Transformers (ViTs) have achieved remarkable success; however, their internal decision-making mechanisms remain largely opaque. The Dynamic Accumulated Attention Map (DAAM) has emerged as a prominent interpretability tool that visualizes model focus by aggregating attention weights across successive blocks. Nevertheless, the fidelity of these maps, specifically the extent to which they accurately reflect the underlying pixels that drive the final classification, is often taken for granted. In this study, we evaluate the fidelity of DAAM using both supervised Data-efficient Image Transformers (DeiT) and self-supervised Self-distillation with no labels (DINO) models under strategic spatial perturbations. Our analysis reveals a critical failure mode in DINO under severe background noise at a Signal-to-Noise Ratio (SNR=−20 dB): while the DAAM visualization paradoxically maintains a sharp and “accurate” focus on the target object, the model’s actual output suffers from total class flipping, shifting from the correct label to an entirely unrelated category. Statistical results on 1000 ImageNet samples confirm this decoupling of visualization and decision influence; DINO’s robustness accuracy plummets to 2.70%, yet its attention maps remain anchored to the salient object. These findings demonstrate that the accumulated saliency provided by DAAM can be a misleading indicator of model logic, as it may highlight regions that the model “observes” but no longer correctly “interprets.” We conclude that interpretability frameworks must transcend spatial heatmaps and incorporate causal fidelity metrics to avoid the pitfalls of deceptive visualization.