Expediting Pre-training for Vision Transformers Through Hierarchical Data Sparsity Optimization

Pre-training cost is a crucial problem in the era of the emerging large models. This is mainly caused by high model computation complexity and the million-scale training datasets, which require extensive time to train from scratch. The substantial training cost associated with recent models poses a significant barrier for individuals and institutions with limited resources. This not only impedes their ability to reach or exceed cutting-edge advancements but also restricts the practical deployment and application of these models on resource-limited devices. Most compression algorithms focus only on efficient fine-tuning or inference from the pre-trained dense models, while time-consuming pre-training is still unavoidable. In contrast, this paper points out that training data is redundant, which is the fundamental reason for the tedious training. To address the issue, this paper aims to introduce sparsity into data and proposes an end-to-end approach for pre-training acceleration by exploiting data redundancy from three sparse perspectives, dubbed Tri-Level ViT++ . Specifically, we leverage a hierarchical reduction scheme by exploring the data sparsity under three levels: the number of training samples in the dataset, the number of patches (tokens) in each sample, and the number of connections between tokens that lie in attention weights. With extensive experiments, we demonstrate that our proposed technique can noticeably accelerate both pre-training and inference for various ViT architectures while maintaining accuracy. Remarkably, under certain ratios, we are able to improve the ViT accuracy rather than compromising it. For example, we can achieve 22.3% speedup with 73.0% (+0.8) Top-1 accuracy on Deit-T, and 22.5% speedup with 80.1% (+0.3) Top-1 accuracy on Deit-S. This proves the existence of data redundancy in ViT.