CrossFuser: Multi-Modal Feature Fusion for End-to-End Autonomous Driving Under Unseen Weather Conditions

Multi-modal fusion is a promising approach to boost the autonomous driving performance and has already received a large amount of attention. Meanwhile, to increase driving reliability under distinct scenarios, it is important to handle unforeseen weather events in the training dataset, which is known as an Out-Of-Distribution (OOD) problem, for autonomous driving algorithms. In this paper, we consider those two aspects and propose an end-to-end multi-modal domain-enhanced framework, namely CrossFuser, to meet the safety orientated driving requirements. CrossFuser first integrates both image and lidar modalities to generate a robust environmental representation through conjoint mapping, elastic disentanglement, and attention mechanism. Further, the perception embedding is used to calculate corresponding waypoints by a waypoint prediction network, consisting of Gate Recurrent Units (GRUs). Finally, the final control commands are calculated by low-level control functions. We conduct experiments on the Car Learning to Act (CARLA) driving simulator involving complex weather conditions under urban scenarios, the results show that CrossFuser can outperform the state of the art.