JFDet: Joint Fusion and Detection for Multimodal Remote Sensing Imagery

Multimodal remote sensing imagery, such as visible and infrared data, offers crucial complementary information that is vital for time-sensitive emergency applications like search and rescue or disaster monitoring, where robust detection under adverse conditions is essential. However, existing methods’ object detection performance is often suboptimal due to task-independent fusion and inherent modality inconsistency. To address this issue, we propose a joint fusion and detection approach for multimodal remote sensing imagery (JFDet). First, a gradient-enhanced residual module (GERM) is introduced to combine dense feature connections with gradient residual pathways, effectively enhancing structural representation and fine-grained texture details in fused images. For robust detection, we introduce a second-order channel attention (SOCA) mechanism and design a multi-scale contextual feature-encoding (MCFE) module to capture higher-order semantic dependencies, enrich multi-scale contextual information, and thereby improve the recognition of small and variably scaled objects. Furthermore, a dual-loss feedback strategy propagates detection loss to the fusion network, enabling adaptive synergy between low-level fusion and high-level detection. Experiments on the VEDAI and FLIR-ADAS datasets demonstrate that the proposed detection-driven fusion framework significantly improves both fusion quality and detection accuracy compared with state-of-the-art methods, highlighting its effectiveness and high potential for mission-critical multimodal remote sensing and time-sensitive application.