Rethinking the Self-Attention in Vision Transformers

Self-attention is a corner stone for transformer models. However, our analysis shows that self-attention in vision transformer inference is extremely sparse. When applying a sparsity constraint, our experiments on image (ImageNet- 1K) and video (Kinetics-400) understanding show we can achieve 95% sparsity on the self-attention maps while main-taining the performance drop to be less than 2 points. This motivates us to rethink the role of self-attention in vision transformer models.