Multisensor Fusion Network for Unstructured Scene Segmentation With Surface Normal Incorporated

Light Detection and Ranging (LiDAR) and camera are two commonly used sensors to acquire data of different modalities in environmental perception. For autonomous vehicles operating in unstructured scenes, it is particularly important to fuse these two different data for semantic segmentation. Most existing methods rely solely on data from a single sensor, which partially limits segmentation performance. Some fusion-based methods, however, are unable to trade off the contribution of point cloud features and image features, resulting in mediocre performance. To address these issues, we propose a multisensor fusion network for unstructured scene segmentation with surface normal (SN) incorporated, called MF-SN Net. For the first factor, we effectively use complementary features to highlight the characteristics of unstructured scenes. Point represented and range view (RV) represented LiDAR information are combined as a baseline to fully utilize 3-D information while ensuring efficiency. RGB images from the camera are reweighted and fused with RV-represented LiDAR features at different scales. Second, we propose a novel method to reweight image features before fusing them into LiDAR features using SN information, which can effectively reduce the negative impact of inaccurate image features. Third, we design a cross-layer attention module to enhance semantic information from high-level features to different layers, which can optimize feature extraction from original point clouds. What is more, we make a synthesized dataset using CARLA simulator to enrich the experimental scenes, so that the network's performance can be evaluated in various conditions. Experimental results on different datasets demonstrate the effectiveness and robustness of our network, showing its competitiveness with state-of-the-art methods.