Answers are grounded in this paper and cite their sources.
With the rapid development of multimodal data fusion based on deep learning, semantic segmentation using the complementary information from the camera and LiDAR sensor can enhance the environmental perception of autonomous driving. However, most fusion methods overlook critical issues arising during the fusion process, including: first, insufficient fusion effectiveness in one-to-one scenarios due to a lack of precise feature alignment; second, low efficiency in the fusion process and a large computational burden on the model. To address these issues, we propose a trans-modal fusion method based on state space model and deformable transformer module. The proposed trans-modal fusion method makes use of the 2-D prior information of the image, such as texture and appearance information. By distilling knowledge from fused 2D–3D representations, it significantly enhances the feature extraction capability of the 3-D point cloud. First, we design a lightweight global gap adjustment attention module based on a linear state-space model, named global gap attention, to better bridge the information gap between point clouds and images. Then, to enhance the efficiency of trans-modal fusion, we adopt a local cross fusion module, which utilizes a many-to-many fusion approach to achieve more accurate feature alignment. Finally, to strengthen the channel information of point cloud features, we introduce an efficient channel attention during the feature extraction process. The experimental results show that the proposed method achieves a better segmentation accuracy compared with baseline. We show superiority of our method on two popular urban datasets SemanticKITTI and nuScenes (achieving mean Intersection over Union (mIoU) scores of 70.9% and 80.4%, respectively).
No takes yet. Share an insight, caveat, or question.
Zhuang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: