Two-Level Hierarchical Adaptive Dynamic Fusion for CNN–LSTM Integration in Fatigue Level Prediction

Driver fatigue is a major contributor to traffic accidents, yet most existing detection systems rely on unimodal inputs or static fusion mechanisms that lack robustness under poor lighting, partially obscured faces, and missing sensor data. This study aims to overcome these limitations by proposing a Hierarchical Adaptive Dynamic Fusion (HADF) model. HADF integrates a two-level adaptive fusion mechanism combining a CNN (ResNet-18) for facial micro-expressions and an LSTM for physiological signals (heart rate, temperature, and accelerometer). The first stage computes adaptive intra-modality weights (α), while the second stage assigns inter-modality weights (γ), enabling context-aware and resilient multimodal integration even under missing-modality conditions. Experiments on a multimodal fatigue dataset show that HADF achieves a validation accuracy of 96.5%, a macro F1-score of 0.96, and ROC-AUC values of 1.00 (Normal), 0.99 (Eye-Closed), and 0.93 (Yawn). Compared with unimodal and static-fusion baselines, HADF improves accuracy by approximately 4.5% and macro F1-score by 6–9%, while maintaining stable performance under incomplete data. These results confirm the novelty of HADF as a two-stage adaptive fusion strategy that enhances accuracy and system robustness, making it suitable for real-time fatigue monitoring in transportation, occupational safety, and healthcare applications.