RVIFNet: Radar–Visual Information Fusion Network for All-Weather Vehicle Perception in Roadside Monitoring Scenarios

Achieving all-weather vehicle perception in roadside monitoring systems composed of cameras and millimeter-wave radars is challenging, primarily due to the ineffective integration of information from these two sources. Particularly in adverse weather conditions, this lack of integration can lead to the monitoring system’s inability to promptly identify and address hazardous situations. However, current fusion methods often have the problem of being dominated by visual information, and they do not sufficiently utilize the complementary aspects of the two-source information. In this article, we present the radar and visual information fusion network (RVIFNet), a novel method that tackles these challenges through enhanced radar data representation and multilevel fusion strategies. First, we develop a pseudoimage representation method for sparse radar data and its feature extraction technique, which enhances its expressiveness and lays the groundwork for feature-level fusion with visual features. Second, we propose a multilevel fusion approach that leverages the complementary attributes of the dual-modal data in terms of spatial localization, resolution, and semantic understanding to achieve fusion at levels of low-level semantics, high-level semantics, and anchor box level. In additiona, we introduce the monitoring perspective for radar and camera (MPRC) dataset, collected and annotated specifically for roadside monitoring scenarios, and elaborate on the spatial–temporal synchronization method for the dual-source data. We evaluate RVIFNet on MPRC and the widely used in-vehicle dataset NuScenes, confirming its effectiveness for all-weather vehicle detection. To best of the authors’ knowledge, this work is among the early attempts to fuse radar and camera data for all-weather vehicle perception in the roadside monitoring scenarios.