Sparsespikformer: A Co-Design Framework for Token and Weight Pruning in Spiking Transformer

As the third-generation neural network, the Spiking Neural Network (SNN) has the advantages of low power consumption and high energy efficiency, making it suitable for implementation on edge devices. However, despite these advantages, SNN still faces accuracy limitations when compared to Artificial Neural Network (ANN). More recently, the most advanced SNN, Spikformer, combines the self-attention module from Transformer with SNN to achieve accuracy comparable to that of ANN. Additionally, to improve the final accuracy, the researchers adopt larger channel dimensions in MLP layers, leading to an increased number of redundant model parameters. To effectively decrease the computational complexity and weight parameters of the model, we explore the Lottery Ticket Hypothesis (LTH) and discover a very sparse (>90%) subnetwork that achieves comparable performance to the original network. Furthermore, we also design a lightweight token selector module, which can remove unimportant background information from images based on the average spike firing rate of neurons, selecting only essential foreground image tokens to participate in attention calculation. Experimental results demonstrate that our co-design framework can significantly reduce 90% model parameters and cut down Giga Floating-Point Operations (GFLOPs) by 20% while maintaining the accuracy of the original model.