Converting Artificial Neural Networks to Ultralow-Latency Spiking Neural Networks for Action Recognition

Spiking neural networks (SNNs) have garnered significant attention for their potential in ultra-low-power event-driven neuromorphic hardware implementations. One effective strategy for obtaining SNNs involves the conversion of artificial neural networks (ANNs) to SNNs. However, existing research on ANN-SNN conversion has predominantly focused on image classification task, leaving the exploration of action recognition task limited. In this paper, we investigate the performance degradation of SNNs on action recognition task. Through in-depth analysis, we propose a framework called Scalable Dual Threshold Mapping (SDM) that effectively overcomes three types of conversion errors. By effectively mitigating these conversion errors, we are able to reduce the time required for the spike firing rate of SNNs to align with the activation values of ANNs. Consequently, our method enables the generation of accurate and ultra-low-latency SNNs. We conduct extensive evaluations on multiple action recognition datasets, including UCF-101 and HMDB-51. Through rigorous experiments and analysis, we demonstrate the effectiveness of our approach. Notably, SDM achieves a remarkable Top-1 accuracy of 92.94% on UCF-101 while requiring ultra-low latency (4 time-steps), highlighting its high performance with reduced computational requirements. The code for our SDM framework will be available at https://github.com/githuberyh/SDM .