Toward Heterogeneous Urban Traffic: Distributed Multi-Agent Reinforcement Learning with Bi-level Intention Inference

Navigating heterogeneous urban traffic environments is challenging for autonomous vehicles (AVs) because of the dense and intricate interactions between AVs, human-driven vehicles (HDVs), and non-motorized vehicles (NMVs). In this paper, we propose a decentralized multi-agent reinforcement learning (MARL) algorithm with a bi-level intention inference module for joint motion and intention prediction of AVs. We model the underlying representation of agents’ intentions on two levels: the high-level intention represents long-term behavioral patterns, while the low-level intention depicts immediate interactive dynamics. By integrating intent-aware motion forecasting, this algorithm ensures the safe and resilient decision making of AV in mixed traffic flow. Experiments are performed in a modified Highway-Env simulation environment, incorporating calibrated models for both HDVs and NMVs based on real-world data. Results demonstrate that, compared with centralized training decentralized execution (CTDE) MARL baseline QMIX, our method yields a 20.0% and 13.8% higher episodic reward in stable and chaotic traffic, respectively, with a 53.2% higher non-collision rate and a 13.8% longer agent lifespan in chaotic traffic. We also compare with a decentralized training and decentralized execution (DTDE) baseline IPPO and demonstrate a higher episodic reward of 7.7% and 15.8% in stable traffic and chaotic traffic, 24.1% higher non-collision rate, and 3.1% longer agent lifespan.