Real-Time Semantic Segmentation for Autonomous Scale Cars using Mixed Real and Synthetic Data

This paper presents a real-time semantic segmentation framework for camera-based environment perception of objects and infrastructure elements in autonomous scale cars. It is specifically targeted towards student competitions such as the Carolo Cup or the Bosch Future Mobility Challenge. To reduce pixel-wise manual annotation efforts, our framework involves a mixture of both synthetic and real image data, carefully tuned towards the unique requirements of the given scenario. Real images are acquired from a 1:10 scale vehicle equipped with a single monocular camera and are manually annotated. Synthetic image data with automatic pixel-wise annotation is obtained via a custom Unity-based simulation pipeline. We evaluate various mixed real-synthetic data strategies to train different state-of-the-art deep neural networks with a focus on both segmentation performance and real-time capability using an NVIDIA Jetson AGX Xavier platform as in-vehicle test bed. Our experimental results show a significant improvement in semantic segmentation performance of the mixed real-synthetic data approach at real-time speeds of approximately 60 FPS on the target platform.