Video Frame Interpolation via Asymmetric Refinement With the Event-Based Reference

This work focuses on leveraging motion cues from event data to address two fundamental challenges inherent in mainstream motion-based video frame interpolation methods (VFI): the occlusion problem and the challenge of estimating non-rigid motion. To this end, we propose a novel framework named DSER++ (Direct Synthesis via Event-based Reference ++) for event-based video frame interpolation (E-VFI). DSER++ avoids estimation errors induced by complex motion patterns through a three-stage purely synthetic process. Particularly, we encode the event data, which contains precise motion trajectories between keyframes, into a sparse reference assisted by the proposed Event-Aware Reconstruction Strategy. Next, we employ the proposed E-PCD module to achieve bidirectional event-guided alignment between keyframes and the reference. Considering the deficiency of events in color and smooth area encoding, the reconstruction for occlusion regions inevitably contains color fluctuations and semantic incoherence. Thus, we introduce a Bidirectional Asymmetric Selective Refinement (BASR) module to targeted optimize such regions of the reconstruction scene by recognizing and providing precise keyframes’ supports. Finally, we equip the multi-scale BASR into an off-the-shelf interpolation refinement network for final predictions. Comprehensive experimental evaluations on both synthetic and real-world datasets underscore the superiority of our approach and its potential to execute high-quality E-VFI tasks. Codes are available athttps://github.com/yuhan0802/DSER-pp