Attention Standard Deviation Based Token Sparsification

Recently, model acceleration techniques have been applied to Vision Transformers (ViTs) to address their high computation costs, particularly in latency-sensitive domains. Advanced techniques primarily focus on token sparsification, aiming to reduce the number of tokens in transformer blocks to accelerate inference. However, they often rely on trial-and-error pruning methods to find a balance between predictive accuracy and pruning efficiency, lacking quantitative characterization. To bridge this gap, this paper introduces an algorithm called Attention Standard deviation based Token Sparsification (ASTS). By utilizing the standard deviation of attention map weights as the threshold for pruning, ASTS removes a specific number of tokens with the lowest attention weights. This approach efficiently adjusts the trade-off between predictive accuracy and pruning efficiency, eliminating the need for extensive exploration in a vast hyper-parameter space. Through extensive experiments on various benchmarks, we empirically validate the effectiveness of the proposed pruning method.