Advanced Driving Assistance Systems (ADAS) & Simulation Data Generation for Autonomous Vehicles

Autonomous vehicles hold transformative potential across society by significantly reducing the human error responsible for the majority of road accidents. Their capacity for consistent, tireless, and data-driven decision-making positions them as a key technology for improving road safety, decreasing traffic congestion, and increasing mobility for underserved populations such as the elderly or disabled. Additionally, autonomous fleets are expected to bring economic benefits through more efficient logistics and transport systems, and environmental benefits through reduced emissions and optimized energy consumption. However, real-world deployment of these systems demands exceptional reliability and generalization to diverse and unpredictable situations—challenges that this thesis directly addresses through architectural innovation and advanced data augmentation strategies. Computer vision and machine learning play a foundational role in autonomous driving, enabling systems to learn complex perceptual and behavioral policies directly from data. Traditionally, autonomous systems were built using modular pipelines consisting of separate stages for perception, planning, and control. However, end-to-end learning has emerged as a compelling alternative, allowing models to map sensory inputs, such as camera frames, directly to low-level control commands like steering and acceleration. This not only simplifies the system design but also reduces error propagation between modules. Within this paradigm, imitation learning has proven effective for learning from expert driving behavior. This work builds upon Conditional Imitation Learning (CIL), a technique where the system learns to replicate human driving behavior based on recorded demonstrations, while simultaneously incorporating high-level navigational commands (e.g., “turn left” or “go straight”) to resolve ambiguous situations. Unlike simple behavioral cloning methods, which often struggle at intersections or complex decision points, CIL introduces conditional branching that tailors the control policy depending on the intended route. This framework enables more structured decision-making and reduces mode collapse caused by conflicting behaviors in diverse driving contexts. Furthermore, CIL provides a scalable foundation for training policies in simulation that can be easily transferred to real-world scenarios. Its modular design also allows it to be integrated with various perception backbones, facilitating experimentation with more advanced vision models such as transformers or multi-view encoders. As such, CIL represents a critical step toward bridging the gap between learned perception and reliable control in autonomous navigation systems. Specifically, it proposes and evaluates an modified instance of CILv3D, which is an enhanced version of CILv2 that incorporates temporal context through image sequences from three front-facing cameras, enabling a more comprehensive and robust understanding of dynamic environments. At the core of this architecture lies Uniformer, a unified spatiotemporal vision transformer model that seamlessly integrates convolutional operations and self-attention mechanisms to capture both local details and global dependencies in driving scenes. The CILv3D model was trained to predict continuous driving controls from input image sequences in an end-to-end fashion. By incorporating temporal context, it overcomes limitations of prior models that rely solely on single-frame input, thereby enhancing performance in scenarios involving motion, such as approaching pedestrians or navigating dynamic traffic. The model benefits not only from the architectural strength of Uniformer but also from the realism of DPGAN-generated training data. Empirical results from multiple benchmarks within CARLA show that CILv3D achieves superior driving accuracy, stability, and collision avoidance compared to earlier CIL versions. Another major contribution of this work is the construction of a diverse, large-scale dataset, that exceeds 160 GB, using the CARLA simulator. This dataset includes a wide range of urban driving scenarios under different weather conditions, traffic patterns, and road geometries. While simulated data is invaluable for safe, scalable experimentation, a common challenge is the sim-to-real gap—the visual and semantic differences between simulated and real-world imagery. To bridge this gap, the thesis leverages a Dual-Pyramid Generative Adversarial Network (DPGAN), a photorealistic image-to-image translation model. DPGAN transforms synthetic driving frames into more realistic ones while preserving semantic structure, significantly improving the generalization capabilities of models trained in simulated environments. The resulting augmented dataset better approximates the visual complexity of real-world conditions, leading to more transferable learned policies. This work also opens the way for future research by setting the foundation for hybrid learning systems that integrate imitation learning with reinforcement learning. Such approaches can first learn from demonstrations and then refine policies through self-improvement in simulation. In conclusion, this thesis demonstrates that integrating transformer-based temporal modeling with high-fidelity sim-to-real augmentation is a promising strategy for developing robust, safe, and scalable autonomous driving systems.