Motion-Oriented Hybrid Spiking Neural Networks for Event-Based Motion Deblurring

Image deblurring based only on the blurry image is challenging as motion information is lost while imaging. Event cameras capture the texture of moving objects in high temporal resolution with asynchronous events. In this paper, we extract motion features from events and fuse them with background features from the image for event-based image deblurring. Spiking neural network (SNN), a widely recognized event feature extractor, is well suited for motion feature extraction due to its high temporal resolution. However, extracting motion information from events exclusively with SNN is challenging. We propose a novel Temporal-local-Spatio Spiking Transformer (TSST) to extract motion intensity and motion attention regions in the spatio-temporal domain. Motion intensity extracted from spiking features is represented as a high temporal resolution motion attention map to guide the fusion of the two networks. In the temporal domain, motion intensity maps spiking features to CNN features as motion features to avoid blurring. In the spatial domain, the motion intensity shows the motion regions and gives the weight of the motion feature during fusion. Moreover, a hybrid feature extraction encoder (HFEE) is introduced, which fully fuses the motion and background features for deblurring. The gradient is back-propagated from CNN to SNN, and the hybrid deblurring network is jointly optimized. We evaluated the performance of our model on the public dataset GoPro and a real event dataset we captured. Codes and pretrained models are available at https://github.com/XDULzx/MotionSNN .