A Distributed Control Method for Urban Networks Using Multi-Agent Reinforcement Learning Based on Regional Mixed Strategy Nash-Equilibrium

Urban network traffic congestion can be caused by disturbances, such as fluctuation and disequilibrium of traffic demand. This paper designs a distributed control method for preventing disturbance-based urban network traffic congestion by integrating Multi-Agent Reinforcement Learning (MARL) and regional Mixed Strategy Nash-Equilibrium (MSNE). To enhance the disturbance-rejection performance of Urban Network Traffic Control (UNTC), a regional MSNE concept is integrated, which models the competitive relationship between each agent and its neighboring agents in order to improve the decision-making process of MARL. The learning rate is enhanced with a self-adaptive ability to avoid a local optimal dilemma; Jensen-Shannon (JS) divergence is utilized to define the learning rate of the modified MARL. A two-way rectangular grid network with nine intersections is modeled via a Cell Transmission Model (CTM). A probability distribution mechanism, which can update the turn ratio of each approach dynamically and discretely, is established to represent the segmented route-decision process of the vehicles. The effectiveness of the proposed control method is evaluated through simulations in the grid network. The results show the influence of major disturbances, such as fluctuation of vehicle arrival rate, fluctuation of traffic demand (e.g. a rapidly rising flow and extreme changes in origin-destination distribution), and disequilibrium of traffic demand (e.g. different arrival flows at each boundary of the urban network), on the performance of the suggested control method. The results can be used to improve the state of the art in order to reduce urban network traffic congestion due to these disturbances.