Occlusion-Aware Driver Monitoring using VLM-Enhanced Situational Understanding

This paper presents a robust, occlusion-aware driver monitoring system (DMS). The system performs driver identification, gaze estimation by regions, distraction detection and face occlusion detection and understanding using VisionLanguage Models (VLMs) to categorize the cause of obstruction (e.g., hand, sunglasses, looking away) under varying lighting conditions. Aligned with EuroNCAP recommendations, the inclusion of occlusion detection enhances situational awareness and system trustworthiness by indicating when the system's performance may be degraded. The system employs separate algorithms trained on RGB and infrared (IR) images to ensure reliable functioning. These algorithms used the multimodal Driver Monitoring Dataset (DMD). We detail the development and integration of these algorithms into a cohesive pipeline, addressing the challenges of working with different sensors and real-car implementation. Evaluation on the DMD and in real-world scenarios demonstrates the effectiveness of the proposed system, highlighting the superior performance of RGB-based models and the pioneering contribution of robust occlusion detection in DMS.