A survey on driving scenario video generation for autonomous vehicle testing: Methodology, datasets, and future prospects

Autonomous vehicle (AV) training and testing require diverse, high-quality driving scenarios. However, real-world datasets are costly, cover safety-critical events sparsely, and require extensive data labeling. Recently, driving scenario generation has become a promising direction to complement data-driven development. This study reviews recent advances across five key paradigms for generating AV driving scenario videos: variational autoencoders (VAEs), generative adversarial networks (GANs), autoregressive models, diffusion models, and world models. The GAN models enhance realism, the VAEs capture multimodal dynamics, the autoregressive models enable long-horizon prediction, the diffusion models provide controllable and high-fidelity simulation of rare events, and world models improve temporal and physical consistency. In addition, large language models (LLMs) are treated as cross-paradigm enablers for controllability and semantics. This study summarizes representative datasets, including the KITTI, nuScenes, and Waymo Open datasets, as well as standard evaluation metrics, such as FID and FVD. Currently, key challenges in the field include computational efficiency and scalability, scene-to-scenario transition, multimodal integration and consistency, and realism with physical fidelity, and the corresponding research opportunities. By integrating generative paradigms, datasets, evaluation metrics, and open challenges into a unified framework, this study provides a useful reference for advancing the safety of autonomous driving testing. It identifies current research gaps and points to opportunities for future development, emphasizing the unique role of video scenario generation in designing robust, reliable autonomous driving systems.