Distributed multi-agent reinforcement learning with neighborhood-aware spatio-temporal coordination for traffic network signal control

Traffic congestion poses a critical impediment to urban economic vitality and environmental sustainability. While deep reinforcement learning (DRL) based traffic signal control (TSC) offers a promising solution, existing multi-agent approaches often rely on static assumptions regarding spatial dependencies or require centralized training with extensive global information, limiting their ability to adapt to dynamically changing traffic conditions. To bridge this gap, we propose a spatio-temporal attention network and neighborhood information sharing-based multi-agent reinforcement learning (SANIS-MARL) framework. The key innovation lies in a synergistic spatiotemporal modeling architecture that explicitly captures the dynamic coupling between temporal traffic evolution and spatial interactions among adjacent intersections. Temporally, we integrate a multi-head attention mechanism with LSTM to characterize the temporal dependencies across multiple time scales. Spatially, a graph attention network v2 (GATv2) adaptively learns the relative importance of neighboring intersections according to real-time traffic conditions. Furthermore, a neighborhood information-sharing mechanism is developed, in which neighborhoods are constructed based on geographic proximity while the influence of neighboring intersections is dynamically weighted through spatial attention. This design enables effective localized coordination and alleviates the non-stationarity problem commonly encountered in multi-agent reinforcement learning. Experiments on an urban network demonstrate that SANIS-MARL consistently outperforms conventional methods and state-of-the-art MADRL controllers in terms of queue length, waiting time, travel time, and emission reduction. Visualization of attention heatmaps further corroborates that the proposed method effectively captures the precursory signatures of backward congestion waves, offering a new paradigm for interpretable and scalable cooperative TSC.