Vision-Based Driver Behavior Recognition using Multi-Modal Deep Learning for Autonomous Vehicles

In this study, a multi-modal driver behavior recognition framework aimed at improving the safety and reliability of autonomous vehicles utilizes sensor and vision data. The framework collects inertial signals (accelerometer and gyroscope) and visual images relating to driver activities to observe driver behaviors. The processing stage begins by preprocessing the inertial data using noise filtering, normalizing and interpolation, followed by feature extraction using Short-Time Fourier Transform (STFT) and Continuous Wavelet Transform (CWT), producing complementary time–frequency representations of the data. For the visual data, begin by contrast enhancement and normalization, before utilizing a Vision Transformer (ViT) to obtain spatial embeddings. The features from both modalities are encoded by a Swin Transformer to capture local and global dependencies, then fused using a combination of cross-attention and channel attention to provide improved interaction of the features. The fused representation is then processed to produce a final behavior representation utilizing ResNet-18, to classify into nine (9) categories. The results of the experimentation show outstanding performance at greater than 99.6% accuracy, F1-score, recall, and precision with a very low false positive and false negative rate, demonstrating its robustness and application for driver monitoring in manner in autonomous systems. The combination of STFT and CWT offers diverse fixed and multi-resolution representations in the frequency domain, whereas integrating ViT and Swin Transformer takes spatial–temporal encoding even further than previous multi-modal methods; the application of ResNet-18 classifier adds even more power to discriminative fusion, making it possible to perform behavior recognition that is more trustworthy than any existing single-or dual-branch systems combined.