Exploring Dual-to-Single-Encoder Transitions for Real-Time RGB-T Semantic Segmentation

RGB-T semantic segmentation improves scene understanding under adverse illumination by exploiting complementary visible and thermal cues. However, conventional intermediate fusion networks preserve two modality-specific encoder branches and repeatedly apply fusion modules throughout the feature hierarchy, increasing the parameter count, computational cost, and inference latency. To address this issue, we propose a dual-to-single-encoder transition strategy that retains modality-specific branches only up to a designated transition stage, applies an explicit attention-enhanced fusion (EAEF) operation at that stage, and processes the remaining encoder stages through a single shared branch. Stage-wise representational alignment is examined using linear centered kernel alignment between independently trained RGB-only and thermal-only encoders, providing a fusion-independent rationale for treating the transition stage as an architectural design variable. Transition stages s0–s4 are evaluated with ResNet-18, ResNet-50, and ResNet-152 backbones on MFNet and FMB. The selected EAEFNet-ResNet-50-s1 configuration reduces parameters and floating-point operations (FLOPs) by 62.6% and 28.5%, respectively, relative to the native EAEFNet-ResNet-50, while improving MFNet mean Intersection over Union (mIoU) by 2.7 percentage points and increasing throughput from 30.4 to 45.9 frames per second (FPS). It also achieves 60.5% mIoU on FMB. These results demonstrate that an early-to-middle encoder transition can preserve competitive segmentation accuracy while providing a favorable accuracy–efficiency trade-off for efficient RGB-T perception.