What question did this study set out to answer?

This study aims to enhance real-time decision-making in autonomous vehicles by using large language models (LLMs) with bird's-eye-view (BEV) perception to reduce inference latency.

March 13, 2026Open Access

Simulation Study on Real-Time Autonomous Driving Decision-Making Using BEV Perception and Large Language Models

Key Points

This study aims to enhance real-time decision-making in autonomous vehicles by using large language models (LLMs) with bird's-eye-view (BEV) perception to reduce inference latency.
Developed an engineering-oriented framework combining BEV perception and low-bit quantized inference.
Used 4-bit post-training quantization (PTQ) for efficient computation.
Conducted experiments on the CARLA simulation platform with various driving scenarios.
Validated findings through 500 independent trials to assess decision-making performance.
Reduced end-to-end inference latency to meet the 10 Hz real-time control requirement.
Improved control quality with lower collision rates compared to traditional methods.
Achieved lower Average Jerk measurements, indicating more stable vehicle control.
Framework demonstrated robustness against environmental degradation and perception noise.

Abstract

Large language models (LLMs) exhibit strong semantic reasoning capabilities for autonomous driving decision-making; however, their substantial inference latency poses a critical challenge for real-time closed-loop vehicle control. This study proposes an engineering-oriented framework to enable latency-constrained LLM-based decision-making by integrating bird’s-eye-view (BEV) structured perception with low-bit quantized inference. The BEV perception module compresses multi-view visual inputs into structured semantic representations, thereby reducing input redundancy and enhancing inference efficiency. In addition, 4-bit post-training quantization (PTQ), combined with an optimized inference engine, is employed to alleviate computational and memory bandwidth constraints during autoregressive decoding. Experiments conducted on the CARLA simulation platform under car-following, overtaking, and mixed driving scenarios—validated through 500 independent trials—demonstrate that the proposed framework substantially reduces end-to-end inference latency while maintaining stable decision-making performance. The results indicate that the system satisfies the 10 Hz real-time control requirement and significantly improves control quality, as evidenced by reduced collision rates and lower Average Jerk compared with both traditional imitation learning (Behavioral Cloning, BC) and the Transformer-based TransFuser baseline. Furthermore, sensitivity analyses confirm the robustness of the framework under environmental degradation and perception noise, underscoring the practical feasibility of deploying LLMs for safe and reliable closed-loop autonomous driving.

Read Full Paperexternally

AI에게 질문

Bookmark

View Full Paper