Augmented Millimeter Wave Radar and Vision Fusion Simulator for Roadside Perception

Millimeter-wave radar has the advantages of strong penetration, high-precision speed detection and low power consumption. It can be used to conduct robust object detection in abnormal lighting and severe weather conditions. The emerging 4D millimeter-wave radar has improved the quality and quantity of generated point clouds. Adding radar–camera fusion enhances the tracking reliability of transportation system operation. However, it is challenging due to the absence of standardized testing methods. Hence, this paper proposes a radar–camera fusion algorithm testing framework in a highway roadside scenario using SUMO and CARLA simulators. First, we propose a 4D millimeter-wave radar simulation method. A roadside multi-sensor perception dataset is generated in a 3D environment through co-simulation. Then, deep-learning object detection models are trained under different weather and lighting conditions. Finally, we propose a baseline fusion method for the algorithm testing framework. This framework provides a realistic virtual environment for device selection, algorithm testing and parameter tuning for millimeter-wave radar–camera fusion algorithms. Solutions show that the method proposed in this paper can provide a realistic virtual environment for radar–camera fusion algorithm testing for roadside traffic perception. Compared to the camera-only tracking method, the radar–vision fusion method proposed significantly improves tracking performance in rainy night scenarios. The trajectory RMSE is improved by 68.61% in expressway scenarios and 67.45% in urban scenarios. This method can also be applied to improve the detection of stop-and-go waves on congested expressways.