SkyEye: Multi-Modal Perception Based Video Stitching for Multi-UAV Surveillance System

Using multiple Unmanned Aerial Vehicles (UAVs) in video surveillance greatly enhances real-time monitoring of large areas. However, images captured by UAVs are separate and limited in view, making stitching crucial for a comprehensive perspective. Current methods combine sensor and visual modalities for video stitching but face challenges in robustness and real-time performance. Environmental disturbances increase errors in sensor and visual data, reducing accuracy and stability, while single-frame stitching incurs high computational costs and slow speeds. To address these issues, we propose SkyEye , a real-time video stitching method for UAV surveillance systems based on multi-modal perception. Specifically, we design a mutual verification method to assess the quality of visual and inertial data. Frames with higher confidence are prioritized for precise stitching. To reduce computational costs, we employ a reference frame scheme that reuses the perspective transformation and feature points of the reference frame for subsequent frames. Meanwhile, we design a video stitching framework based on a per-frame parallel unit to speed up the algorithm execution. We have implemented a prototype of SkyEye and carried out extensive evaluations. Experiment results show that SkyEye outperforms state-of-the-art methods, improving speed by 66.1% and accuracy by 28.2%.