TL-4DRCF:A two-level 4D radar-camera fusion method for object detection in adverse weather

In autonomous driving systems, cameras and Light Detection and Ranging (LIDAR) are two common sensors for object detection. But both sensors can be severely affected by adverse weather. With the development of radar technology, the emergence of the 4D radar gives a more robust solution for sensor fusion strategies in 3D object detection tasks. This study proposes a two-level 4D radar and camera fusion model called TL-4DRCF, which performs a two-level fusion of 4D radar and camera information at the data and feature levels. In the data-level fusion stage, the radar point cloud is projected onto the image and fed as additional information to the image into the EarlyFusion-Net (EF-Net), which is the network designed for simultaneous extraction of point cloud and image features. In the feature-level fusion stage, the Radar-Camera Alignment (RCA) module is proposed to accurately correlate point cloud voxels and pixel-level image features while consuming less inference time. The correlated features are used to predict the class and location of the object through a standard 3D detection framework. The proposed TL-4DRCF was validated on the View-of-Delft (VoD) dataset and the VoD-Fog dataset performed by artificial fog processing. The experimental results show that the proposed model outperforms the baseline method PointPillars on the VoD dataset by 3.8% mAP and the LIDAR-camera-based method MVX-Net in the driving corridor area of the VoD-Fog dataset by 0.39% mAP.