Object Detection With Spiking Neural Networks on Event Videos for Driving Scenarios

Event cameras excel in imaging objects under challenging lighting conditions with low latency. Their sparse and asynchronous event outputs closely align with the operational paradigm of Spiking Neural Networks (SNNs). An SNN mimics biological neurons of brain by transmitting information via binary spikes. The combination of biologically inspired event-based sensing with neuromorphic computing, while individually promising, faces significant challenges in their coherent integration to deliver better performance than existing solutions for real-time multi-scale, multi-rate object detection problems. In this paper, we propose a new Spike Tensor-based Multi-scale Object Detector (STMOD) for event video analytics in driving scenarios. STMOD features a cross-attention spatio-temporal spikes generator to convert asynchronous events into a high-dimension binary vector, a spiking backbone with three pyramid feature levels to fuse multi-scale spatio-temporal features and an asymmetric decoupled detection head to achieve high precision, energy efficient and responsive object detection without compromising the temporal resolution. Additionally, a new training method is proposed to overcome the mismatch in SNN’s state between training with independent samples and testing with video streaming to improve its performance on continuous video feed without impacting inference time resolution. Experimental results on the GEN1 and DSEC-Detection datasets show that STMOD achieves state-of-the-art mAP of 44.3% and 34.2%, respectively, with an inference time resolution of 5 ms. Finally, it also rivals existing event-based object detectors with significantly lower number of parameters and higher energy efficiency.