Transformer-based Ensemble for Robust Driver Eye State Recognition in Smart Vehicles

Smart vehicles increasingly employ artificial intelligence to enhance driving safety and performance. Continuous monitoring of the driver is vital to prevent accidents and ensure safe operation. Vision-based sensing provides a non-invasive means to assess driver behavior, alertness, and attention. Reliable and real-time detection of drowsiness is essential to support higher levels of vehicle automation. Classical vision-based approaches that relied on eye landmarks and handcrafted ratios often struggle under challenging conditions such as poor illuminations, variations in head movements, and occlusions. Although Convolutional Neural Networks (CNNs) have significantly improved detection accuracy, they still face limitations in modeling long-term dependencies and generalizing across diverse driving environments. Eye state (open vs. closed) serves as a fundamental visual cue for blink detection, attention monitoring, and higher-level drowsiness estimation. Most existing studies and publicly available datasets predominantly provide drowsy or alert labels at the frame or segment level based on full-face without direct supervision on the observable cue of eye state. In addition, in certain situations or for cultural reasons, full-face images may not be available due to religious or privacy considerations, leaving only the eye region visible. In such cases, accurate eye-state detection becomes particularly crucial. To address these challenges, we propose an ensemble framework for driver eye-state detection in smart vehicles. The architecture integrates three transformer encoders namely ViT, DeiT, and Swin-T. We preprocessed the original UTA-RLDD and NTHU-DDD video dataset and generated eye-state annotations for the extracted eye regions using Eye Aspect Ratio (EAR) computation. The model was evaluated on MRL-Eyes, UTA-RLDD-Eye, and NTHU-DDD-Eye state annotations. Experimental results demonstrate promising and consistent performance for the proposed ensemble model across all eye-state conditions and datasets.