Network Pruning with Gradient Smoothing

Deep Neural Networks have found widespread useacross various domains, demonstrating notable success. Nonethe-less, their inherent complexity, stemming from the inclusion ofmillions of parameters, presents challenges during deployment insystems requiring low latency. Consequently, there’s a growingneed to develop lightweight neural networks that deliver compa-rable performance during inference. This study introduces a novelweight-based pruning method, wherein weights are graduallypruned based on their momentum from previous iterations. Eachlayer of the neural network is assigned an importance valuedetermined by both its relative sparsity and the magnitude of itsweight in prior iterations. The effectiveness of our approach isassessed on popular network architectures like AlexNet, VGG16,and ResNet50, employing image classification datasets CIFAR-10and CIFAR-100. Results indicate superior performance in termsof both accuracy and compression ratio compared to existingmethods. Notably, our method achieves a compression rate of15× while maintaining the same level of accuracy degradationacross both datasets