Pre-Trained Bidirectional Temporal Representation for Crowd Flows Prediction in Regular Region

Deep learning-based methods for predicting spatial-temporal data such as crowd flows need to consider both spatial dependency and temporal dependency. Previous research mainly focused on modeling spatial dependency, whereas studies on temporal dependency are few. Existing finite deep learning-based methods for temporal dependency modeling can be divided into RNN-based methods and domain knowledge-based methods. However, RNN-based methods are hard to learn very long-term temporal dependency, and domain knowledge-based methods cannot model temporal dependency automatically, depending on data pre-processing based on prior knowledge. In view of the problem, crowd flows prediction in regular gridded regions are studied and a model called Pre-trained Bidirectional Temporal Representation (PBTR) based on Transformer encoder is proposed capable of modeling very long-term temporal dependency automatically. PBTR is simple, scalable, and can be combined with any other spatial component. Furthermore, we introduce Crowd Flows Prediction based on PBTR (CPPBTR) to form a Transformer based encoder-decoder framework. There are two decode stages in the proposed model. At decoder-stage 1, `draft' sequence is generated. At decoder-stage 2, each timestep of the `draft' sequence is masked and fed into PBTR to predict the refined flow for each masked position. Experiment results demonstrate that our method outperforms RNN-based methods and domain knowledge-based methods.