Light-Aware Modality Balancing Network for Multimodal Pedestrian Detection

The visible light and infrared thermal multimodal images in autonomous driving provide a wealth of information for pedestrian detection, and its challenge lies in utilizing the complementary information across modalities to obtain an optimal joint representation. This study proposes a light-aware modality balancing network (LMB-Net) for pedestrian detection by fusing visible light and infrared thermal images. We designed an alignment complementary fusion module across modalities to exchange target information. Deformable convolutions are employed to automatically perform spatial deformation on features, thereby eliminating perception biases caused by misalignment. Furthermore, as the contribution of different modalities to pedestrian detection varies under different lighting conditions, we designed a light-aware module to utilize the distinct advantages of visible light and infrared thermal images. Extensive experiments on the KAIST and LLVIP datasets demonstrate that our method achieves the best detection performance compared to some other methods.