Multimodal Fusion of Visual and Gyroscopic Data for Road Surface Monitoring Using ResNet50-Based Transfer Learning

Driven by climate change and escalating traffic, rapid road deterioration threatens public safety, prompting a shift from costly manual inspections toward scalable solutions. High-precision laser scanners offer accuracy, but prohibitive costs limit continuous monitoring across extensive road networks. To overcome these constraints, this study proposes an automated multimodal monitoring system integrating real-time video imagery with gyroscopic impact data. The architecture employs a late fusion approach, combining visual features extracted from the Global Average Pooling (GAP) layer of a pre-trained ResNet50 model with synchronized gyroscopic time-series data. Experimental results revealed that while the initial baseline model achieved a recall of 0.75 and an F1-score of 0.55, applying data balancing enhanced performance, increasing the recall to 0.98 and the F1-score to 0.65. When evaluated on a geographically independent highway section, the model identified 44 bad road segments and encompassed 94% of actual bad segment durations, predicting a total duration of 741 s compared to the ground-truth 564 s. Ultimately, despite a tendency for over-detection, the model demonstrates significant practical utility as a conservative decision-support tool for proactive road maintenance due to its high sensitivity and robust detection capabilities.