PMDet: Patch-Aware Enhancement and Fusion for Multispectral Object Detection

Multispectral object detection addresses the limitations of single-modal approaches by fusing complementary information from visible and infrared images, thereby improving robustness in complex environments. However, the inter-modal representations are inherently misaligned due to sensing discrepancies, and the complementary cues they provide are often imbalanced, making it difficult to exploit modality-specific information effectively. Moreover, directly merging features from different modalities can introduce noise and artifacts that deteriorate the detection performance. To this end, this paper proposes a patch-aware enhancement and fusion network for multispectral object detection (PMDet). This method employs a dual-stream backbone equipped with the patch-aware Feature Enhancer (FE) module for cross-modal features alignment and enhancement. FE not only reinforces the feature representation of key regions but also helps to suppress local noise and enhance the model’s perception of fine textures and differences. Building on these enriched features, the patch-based Feature Aggregator (FA) module allows for efficient inter-modal feature interaction and semantic fusion with noise resistance. Specifically, both FE and FA modules leverage the shifted-patch design to preserve computational efficiency while enabling long-range modeling. In this regard, PMDet couples multi-scale cross-modal semantic enhancement with deep semantic fusion to form a stable and discriminative multimodal representation pipeline. Experiments on FLIR, LLVIP, and VEDAI demonstrate that the method outperforms mainstream approaches in detection accuracy and robustness, and ablation studies further verify the effectiveness of each module.