MSCT-Net: A Multi-Stream CNN–Transformer Network for Driver Drowsiness Detection

Driver drowsiness is a major cause of fatal car accidents, and camera-based monitoring offers a promising way to detect it. Detecting drowsiness from video requires modeling both facial appearance and its temporal evolution. In this article, we propose MSCT-Net, a multi-stream CNN–Transformer network for this purpose. Region-specific CNN backbones encode the face, right eye and mouth in every frame. Then, a single Transformer encoder jointly models their temporal dynamics across regions and time. MSCT-Net is trained from scratch on the challenging NTHU Drowsy Driver Detection dataset, which covers daytime and nighttime driving with and without eyewear. The architectures for the backbones and the Transformer are selected experimentally on the validation set. Compared with a baseline that shares the same backbones but averages features over time, MSCT-Net improves the clip-level accuracy from 78.70% to 82.16%, the F1-score from 81.15% to 83.15% and the AUC from 0.8807 to 0.9052, with higher accuracy and AUC in every scenario of the dataset. It also outperforms a CNN-LSTM baseline of similar size (accuracy 80.94%, F1-score 82.79%, AUC 0.8983). On the official test set of 14 subjects, MSCT-Net achieves a frame-level accuracy of 82.26% without any post-processing.