Enhancing robustness in 3D object detection through camera-radar fusion with consideration for radar variability

To ensure safe driving with autonomous vehicles, the integration of multiple sensors is highly recommended to enhance the robustness and accuracy of object detection. Despite extensive research on the fusion of camera and millimeter-wave radar methods, challenges remain in achieving sufficient robustness and comprehensive data integration. In this paper, we focus on the middle-level fusion of camera and radar features to exploit the effectiveness of a combination using these two sensing modalities. A hierarchical framework is proposed to facilitate middle-level data fusion from different modalities. Our framework incorporates a primary regression head that estimates the objects’3D bounding box including the position and orientation. Additionally, we introduce a novel radar feature extraction strategy to address the crucial data association issue, effectively associating radar points with their corresponding objects. With considering the different information layers and probability distribution of radar points, we utilize Gaussian heatmaps as an extra channel to generate feature maps for the radar. Furthermore, for the radar information missing and mismatching caused by the soft time-synchronization, we adopt an Interacting Multiple Model (IMM) based states estimating method to optimize key-frames for reliable detection in real-world applications. Finally, we evaluate our framework on the nuScenes dataset, which shows the 3.08% improvement in NDS-score compared with the representative camera-radar fusion method. Moreover, comprehensive comparisons have been conducted on an unmanned ground vehicle (UGV) platform in real-world scenarios, showcasing the accuracy and real-time performance of the proposed approach.