Adaptive Cross-Modal Fusion with Temporal Modeling for Near-Real-Time Driver Drowsiness Detection on Embedded Systems

Drowsiness at the wheel is a major factor in road accidents, requiring reliable detection systems. Existing vision-based approaches generally rely on unimodal features or static fusion strategies, limiting adaptation to inter-subject variability and changing acquisition conditions. This article proposes a multimodal hybrid architecture combining a lightweight convolutional backbone (MobileNetV4) with biometric descriptors (EAR, MAR) from MediaPipe FaceMesh. A cross-modal fusion module integrates bidirectional cross-attention with an adaptive gate that dynamically weights each modality's contribution based on learned feature patterns, improving cross-subject robustness under the strong domain shift of NTHU-DDD, though a simpler concatenation baseline (BiLSTM with attention) remains competitive, and in one case superior, on the less heterogeneous YawDD dataset. Controlled perturbation experiments show the gate's adaptivity does not equate to an explicit reliability estimate. A Transformer Encoder then models temporal dependencies over the fused representation. The approach is evaluated on YawDD and NTHU-DDD, including a shared-backbone LOSO ablation-control protocol and a strictly leakage-free LOSO evaluation for the S006 fold, together with cross-dataset generalization analysis. Configuration E achieves 91.51% ± 8.89% accuracy and 91.39% ± 9.17% F1-score on NTHU-DDD under the shared-backbone ablation-control protocol; the strictly leakage-free S006 fold achieves 76.35% accuracy. A stricter zero-shot evaluation reveals a substantially larger drop to 55.93% F1-score, highlighting cross-dataset generalization as an open challenge. Deployment on Raspberry Pi 5 confirms near-real-time feasibility (∼106 ms/frame, 9.45 FPS), compatible with the model's 1-2 second decision window. A Grad-CAM analysis is suggestive of the physiological relevance of the regions exploited by the model.