Proposal-Guided Multi-Scale Radar and Vision Fusion for 3D Object Detection

Radar and vision fusion in autonomous driving has gained increasing attention, however, most existing feature-level fusion methods (named as feature map-based fusion) typically map sparse point clouds into dense feature maps, fuse them with visual features, and then apply the combined features to detect sparse objects. These methods face challenges due to the structural disparity between sparse point clouds and dense feature maps, and memory-redundant mapping of point cloud processing. This paper proposes a proposal-guided radar and vision fusion framework for 3D object detection in autonomous driving, where proposals are referred to as potential objects with abundant high dimensional features. Specifically, two frame radar point clouds are first accumulated by deep learning-based motion compensation with supervision. Then, proposals are generated from radar point clouds and vision images. Subsequently, radar and vision fusion is performed on the proposal level using attention mechanism. Finally, objects are detected based on the fused proposal feature. The proposed framework introduces an object-centric feature fusion methodology that skips the common process of point-to-feature map projection, thereby circumventing the inherent structural disparity between sparse point clouds and dense feature maps. Experiments on two different autonomous driving datasets (nuScenes and VoD) demonstrate that our fusion method can outperform most published radar and vision fusion networks. The source code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/leehungu/prv</uri>