Automating gaze-to-object mapping in driving simulators using machine learning and eye tracking

Driving is highly dependent on visual cues; scanning patterns therefore offer critical insights when analyzing interaction with such technologies and driving performance. However, such analyses are time-consuming and so have not been used extensively. To overcome this barrier, we propose to automate gaze-to-object mapping in a driving simulator environment, leveraging established object detection algorithms and eye tracking technology to automatically determine which objects in the road environment a driver is looking at. Images from a driving simulator were manually annotated to identify relevant road objects and used to train a machine learning object detection algorithm (Faster R-CNN). The object detection algorithm was then used to create annotated videos. These were correlated with eye tracking data to provide lists of foveated objects with timestamps as a pilot study to confirm concept viability. This gaze-to-object mapping was compared to driver narration of their visual attention to evaluate model performance. The automated system achieved an overall precision of 1.00, recall of 0.84, and F1 score of 0.91. The class-specific metrics highlighted variability in object detection performance, with classes containing large objects with low intra-class variance (e.g. buses) being the most accurately detected, while classes containing small disparate objects (e.g. road signs and pedestrians) had the lowest detection accuracy. The mean average precision across all classes was calculated to be 0.84. This study serves as a framework and proof of concept for automating gaze-to-object mappings within driving simulator environments. The promising results support testing with participants to further assess the system’s accuracy and feasibility.