Smart city decision-making increasingly relies on heterogeneous urban data sources. Dense traffic sensor streams provide continuous quantitative measurements, while citizen-generated textual reports offer event-driven contextual information. However, integrating these modalities remains challenging due to temporal misalignment, textual sparsity, and semantic noise. This paper investigates multi-modal learning for traffic congestion severity prediction through an experimental integration of open traffic sensor data (METR-LA: Los Angeles, USA) and citizen-generated textual reports (NYC 311: New York City, USA). Congestion severity is formulated as a four-class classification task derived from traffic speed measurements. We propose an end-to-end framework that combines: (i) sensor time-series encoding using a GRU-based temporal encoder, (ii) textual representation learning using a BERT-based encoder, (iii) a symmetric time-window alignment strategy (±Δ) to associate irregular reports with sensor time steps, and (iv) multiple fusion architectures, including early fusion, late fusion, and a cross-attention module for cross-modal interaction modeling. Experiments on publicly available datasets show that multi-modal early fusion achieves the best overall performance (Accuracy = 0.8283, Macro-F1 = 0.8231) compared to uni-modal baselines. In the studied cross-city setting with sparse and weakly aligned textual signals, the proposed cross-attention fusion does not outperform the strong sensor-only baseline, suggesting that the sensor modality dominates when cross-modal signal strength is limited. These results highlight both the potential and the practical constraints of multi-modal fusion in heterogeneous smart-city environments, emphasizing the importance of alignment design, modality relevance, and transparent experimental validation.
Nouf Alkhater (Thu,) studied this question.