VANETs Meet Autonomous Vehicles: A Multimodal 3D Environment Learning Approach

In this paper, we design a multimodal framework for object detection, recognition and mapping based on the fusion of stereo camera frames, point cloud Velodyne LIDAR scans, and Vehicle-to-Vehicle (V2V) Basic Safety Messages (BSMs) that are exchanged using Dedicated Short Range Communication (DSRC). We merge the key features of rich texture descriptions of objects from 2D images using Convolutional Neural Networks (CNN). In addition, depth and distance between objects are provided by the 3D LIDAR point cloud and the awareness of hidden vehicles is achieved from BSMs' beacons. We present a joint pixel to point cloud and pixel to V2V correspondence of objects in frames of driving sequences in the KITTI Vision Benchmark Suite. We achieve this by using a semi-supervised manifold alignment approach to achieve camera-LIDAR and camera-V2V mapping of their recognized persons and cars that have the same underlying manifold.