3D-FAD: Informativeness-guided Multimodal 3D Object Detection for Federated Autonomous Driving

Autonomous driving has been adopted in various applications, including taxis, logistics, and agriculture, to address labor shortages and enhance safety. A critical component is 3D object detection, relying on deep neural networks (DNNs) that integrate LiDAR and camera data for robust road planning. Federated Autonomous Driving (FAD) utilizes Federated Learning (FL) to train these DNNs, addressing the demand for diverse training datasets while preserving client privacy. However, extending the existing FAD framework to 3D object detection remains challenging due to insufficient consideration of modality heterogeneity and informativeness. The paper introduces 3D-FAD , a pioneering informativeness-guided multimodal FAD framework. 3D-FAD is the first to explicitly integrate modality informativeness into multimodal 3D object detection for FAD by a convolution-based encoder. It utilizes this informativeness for efficient client selection and aggregation at the server. Furthermore, 3D-FAD introduces Masked Modality AutoEncoders to impute deficient LiDAR features for clients with limited LiDAR FOV and low-cost cameras in real-world autonomous driving. Our experiments on real-world autonomous driving datasets nuScene and Waymo show that 3D-FAD outperforms baselines by up to 34.1% and 39.2% in key metrics, along with better performance across different object classes and environments.