Behavior-Constrained Multi-Agent Proximal Policy Optimization for Cooperative Vehicle Control at Future Intersections

Abstract Autonomous driving technology faces significant safety challenges, particularly at unsignalized intersections. Centralized cooperative methods have been developed to manage the flow of connected and automated vehicles. However, many existing approaches depend on basic control algorithms, leading to lengthy inference time and suboptimal solutions. Consequently, real-time performance, road resource utilization, and traffic efficiency are compromised. While some studies have integrated reinforcement learning (RL) techniques to address these issues, they often comprise safety due to reward-driven optimization and the oversimplification of traffic scenarios, such as designing specific flow directions. These limitations raise concerns about their real-world applicability and safety. To address these shortcomings, this paper introduces a novel behavior-constrained proximal policy optimization (BCPPO) method for RL-based cooperative vehicle control at intersections. First, the problem is formulated as a multi-agent RL task within a Markov Game (MG) framework. A multi-agent proximal policy optimization (MAPPO) algorithm is proposed to handle the complex cooperative dynamics among multiple agents. The policy network employs a Long Short-Term Memory (LSTM) encoder to capture extensive social interaction information among the agents. Second, intersection control problem is formalized within the MG framework, and a safety-enhanced cooperative vehicle control strategy, BCPPO, is proposed. This method integrates formal safety verification and behavior constraints into the training and deployment of MAPPO to ensure safety and robustness. Finally, extensive simulation experiments are conducted across various intersection scenarios to evaluate the performance of BCPPO against RL-based proximal policy optimization (PPO), the rule-based first-come-first-served (FCFS) method, and the optimal control (OC)-based vehicles-intersection control system (VICS). The results demonstrate that BCPPO achieves a zero-collision rate during deployment and enhances driving comfort by 60.75%, compared to the non-safety-aware PPO method, which has a collision rate of about 13.85%. Furthermore, BCPPO improves traffic efficiency by 16.15% in comparison to FCFS and reduces inference time by a factor of 71.73 relative to the VICS method.