Exploring data sparsity correlation: boosting DNN training on the edge via sparsity modeling

Exploiting sparsity to eliminate redundant computation and memory overhead is essential for enhancing the efficiency of on-edge deep neural network (DNN) learning. The sparsity features leveraged by existing works are derived from empirical or statistical analyses and are focused on specific phases of training. As a result, the available sparsity features are restricted to the model sparsity of convolution operations. This limitation prevents the full exploitation of the advantages of sparsity, thereby constraining the potential for efficient edge learning. This paper observes that data sparsity correlation exists in the zero-value positions between different operands during training, which can be efficiently leveraged. We propose a unitwise analytic approach that investigates data sparsity correlation by exploring the zero propagation rules among different operands. Based on this, we construct a sparsity propagation model that formally describes these features and proposes a sparsity exploitation method to ensure sufficient sparsity exploitation. To our knowledge, this paper is the first work to model sparsity during training. We also present an energy-efficient sparse training accelerator (ESPACE) that enables multiple sparsity features to avoid unnecessary computations and memory footprints, alleviating the contradiction between training high-demand and resource-constrained edge devices. Extensive experiments validate that ESPACE achieves speedups of up to 3.2 \(× \) , 1.69 \(× \) , and 1.73 \(× \) , and energy efficiencies of up to 2.8 \(× \) , 1.75 \(× \) , and 1.9 \(× \) , compared to the dense accelerator and two other sparse training accelerators, respectively. The proposed sparsity-exploitation method can be used alongside prior dynamic pruning methods during training to achieve a 1.89 \(× \) speedup and a 1.59 \(× \) improvement in energy efficiency, even though these approaches leverage model sparsity for aggressive pruning. These results demonstrate that modeling and exploiting sparsity correlation during training can significantly outperform conventional sparsity strategies that rely solely on model sparsity, providing a more general and effective pathway for improving both computational efficiency and energy efficiency in on-edge learning systems.