3D perception algorithm of unstructured environment based on point cloud enhanced pixel fusion

Based on the complementary and enhanced fusion of 3D point clouds and 2D RGB images, this paper designs an end-to-end learning framework—Point Cloud Enhanced Depth Pixel Fusion Network (PEPF-Net), aimed at enabling robots to achieve accurate 3D perception of unstructured environments. In the process, we address four key problems in 3D perception tasks: enhancing RGB representation using the reflection intensity and depth information of point clouds to generate Depth-RGB Pixel (D-Pixel); proposing Point-by-Point Vector Attention (PVA-Net) to model the vector relationships of point clouds, to obtain deep-level point cloud features, and to achieve direct and effective fusion of heterogeneous data; designing a Layered-Transformer (L-TsfmNet) feature extractor to hierarchically extract D-Pixel features; proposing Variable Window Self-attention (VS-a) to focus on the relationships between local “window tokens” and avoid the complexity of global computation. Extensive experiments on the KITTI dataset demonstrate that PEPF-Net outperforms the currently common advanced environmental 3D perception algorithms.