Implicit-information Embedding and Enhanced Spatial-Temporal Multi-Head Attention Transformer for Traffic Forecasting

Abstract Traffic flow prediction is a critical challenge in the field of intelligent transportation systems, as the accuracy of future traffic flow predictions directly impacts the efficiency of urban traffic systems. However, existing deep learning prediction models face several issues. Firstly, many methods concentrate solely on traffic flow data for embedding, overlooking the implicit information underlying these data, which includes human behavioral trends, traffic patterns within and around communities, urban weather conditions, semantic information, and temporal periodicity. Secondly, methods that employ the original multi-head self-attention mechanism calculate attention scores point by point in the temporal dimension without leveraging context, resulting in less accurate attention computation. Thirdly, existing methods struggle to handle both short-range and long-range spatial dependencies simultaneously in the spatial dimension. To address these issues, this paper proposes a method called IEEAFormer (Implicit-information Embedding and Enhanced Spatial-Temporal Multi-Head Attention Transformer). IEEAFormer introduces an embedding layer to capture implicit information in the input, replaces the traditional multi-head self-attention with Temporal-Environment-Aware self-attention in the temporal dimension to enable each node to perceive its contextual environment, and employs two unique graph mask matrices in the spatial dimension to capture both long-range and short-range dependencies. Validation results on four real-world traffic datasets demonstrate that the proposed IEEAFormer outperforms most comparable models in prediction performance.