Rule-Aware Multi-Agent Reinforcement Learning for Cooperative Decision-Making in Heterogeneous Traffic

Efficient and safe decision-making in heterogeneous traffic environments—particularly on highways with mixed autonomy—remains a significant challenge for autonomous driving systems. Existing deep reinforcement learning (DRL) approaches often struggle to adapt to dynamic and stochastic traffic conditions, especially when managing complex maneuvers. To address these limitations, we propose a Rule-Aware Multi-Agent Soft Actor-Critic (RW-MASAC) framework, which decouples the decision-making process into longitudinal and lateral subtasks handled by specialized agents within a collaborative policy structure. The integration of rule-aware guidance accelerates policy convergence, improves decision rationality, and enhances adaptability under heterogeneous traffic flows. Extensive experiments conducted in randomly generated heterogeneous highway scenarios across varying traffic densities demonstrate that RW-MASAC consistently achieves higher success rates, lower collision frequencies, and more stable speed profiles compared to existing baseline methods, highlighting its robustness and efficiency in complex mixed-traffic environments.