Masked Token Enabled Pre-Training: A Task-Agnostic Approach for Understanding Complex Traffic Flow

Accurate analysis of traffic flow (TF) data is crucial for the vehicular applications. Conventional deep learning models require task-specific training and are susceptible to high-frequency disturbances, degrading the feature representation capability. To overcome these limitations, this paper proposes a Token-based SelfSupervised Network (TSSN) that can learn TF features in both tokenization and task-agnostic manners. It provides a properly bootstrapped pre-training model for various downstream tasks. In support of the edge computing and vehicular cloud computing, the pooled computational resources facilitate real-time inferences of downstream models. In TSSN, TF data are segmented into tokens. A pretext task, named as Masked Token Prediction (MTP), is then developed to allow TSSN to understand the underlying correlations of TF by predicting randomly masked tokens. By utilizing MTP, TSSN is able to extract the high-level intrinsic semantics of TF, and provide general-purpose token embeddings, leading to improved overall performance and enhanced ability to adapt to different tasks. By substituting the last fully-connected layers with a group of untrained new layers and fine-tuning using small-scale task-specific data, TSSN can be utilized for a variety of downstream tasks in vehicular applications. Simulation results indicate that the TSSN enhances overall performance in comparison to state-of-the-art models.