MariSense: A multi-modal dataset for environmental perception in maritime autonomy

Maritime autonomous navigation enables unmanned surface vehicles to perform multiple tasks and can also serve as a driver-assistance system for manned vessels, improving safety and comfort. Compared with on-road scenes, maritime perception faces unique challenges: optical sensors degrade due to reflections and sea fog, while wind and waves cause large platform motions that disrupt perception stability. Therefore, high-quality multi-modal datasets are crucial. Meanwhile, 4D millimeter-wave (mmWave) radar has been successfully applied in on-road scenes due to its all-weather, all-day, and high-resolution sensing capabilities, but its maritime characteristics and waveform design effects remain underexplored. To address these challenges, we present MariSense, a multi-modal dataset for maritime autonomy perception. MariSense is the first to introduce the concept of dynamic waveform 4D mmWave radar into maritime scenes, integrating multi-waveform radars with visible and thermal cameras, LiDAR, marine radar, Automatic Identification System (AIS), and navigation sensors, all with extrinsic calibration. The dataset covers various scenes such as anchorages and fairways under varying weather, time, and sea conditions. We establish benchmarks for object detection, tracking, and odometry, and evaluate baseline methods, highlighting the need for improved maritime perception, multi-modal fusion, and adaptive 4D mmWave radar waveform design. The dataset can be downloaded from https://github.com/ORCAUBOAT/MariSense .