A pedestrian detection method based on feature fusion of image and point cloud

Abstract Stable and reliable perception capability is the basis for the safety of autonomous driving, and pedestrian detection is one of the key tasks for vehicle-mounted sensors to perceive the environment. In order to make full use of the complementarity of vehicle cameras and lidars, we make improvements on the basis of the EPNet algorithm, and a pedestrian detection method based on pre-fusion of point cloud and image data is proposed. Use the bidirectional cascaded feature fusion module to achieve more information exchange between image and point cloud data, and obtain more comprehensive fusion features; design a consistency loss function to enhance the correlation between location confidence and category confidence and improve model detection accuracy. Validated on KITTI and other datasets, the detection result of pedestrians can reach 84% mAP, 4.49% higher than the EPNet on difficult pedestrian samples. Compared with a single visual sensor, the proposed method has a better detection effect on objects affected by shadow or longer distance. Finally, the model is accelerated based on the TensorRT custom plug-in and uses CUDA to improve the efficiency of multimodal data pre-processing and post-processing. Deployed on the Nvidia Jetson Orin edge computing device, the model runs at 10 frames per second, and the inference speed is increased by about 60%, laying the foundation for the application of algorithm engineering.