A Survey on Spatio-Temporal Prediction: From Transformers to Foundation Models

Spatio-Temporal (ST) data is pervasive on the various aspects in our daily lives. By mining the ST information from the data, we are able to predict trends in numerous domains. The Transformer, and one of its more recent enhancements, foundation models, have achieved a remarkable success in such ST prediction. In this paper, we first survey the state of the art of Transformers-related work, then introduce the network architecture of the Transformer and summarize the improvements to adapt to the ST prediction Transformer and foundation models, including module enhancement and adjustment. Subsequently, we categorize the ST Transformer and foundation models in selected applications in some relevant domains, mainly urban transportation, climate monitoring, and motion prediction. Next, we propose an evaluation method in the ST prediction with Transformers and foundation models, list the most relevant open-source datasets, evaluation metrics and performance analysis. Finally, we discuss some future directions on the task of ST prediction with Transformer and foundation models. Relevant papers and open-source resources have been collated and are continuously updated at: https://github.com/cyhforlight/Spatio-Temporal-Prediction-Transformer-Review.