Key result
Machine learning with time-on-task labeling achieves ~99% mental fatigue estimation accuracy versus behavioral performance labeling.
Why the study?
Mental fatigue detection had seldom been evaluated in realistic visual search tasks, and time-on-task versus performance-based labeling metrics had not been directly compared.
Does time-on-task labeling compared to behavioral performance labeling improve the classification accuracy of mental fatigue detection in participants performing a visual search task?
Does time-on-task labeling compared to behavioral performance labeling improve the classification accuracy of mental fatigue detection in participants performing a visual search task?
Absolute Event Rate: 99.3% vs 52.2%
High classification accuracies for mental fatigue using time-on-task labels do not translate to operationally relevant behavioral performance metrics.
High ML accuracy with time-on-task labels fails to predict behavioral performance; leaves open operational utility of fatigue detection models.
Objective : A central component of search and rescue missions is the visual search of survivors. In large parts, this depends on human operators and is, therefore, subject to the constraints of human cognition, such as mental fatigue (MF). This makes detecting MF a critical step to be implemented in future systems. However, to the best of our knowledge, it has seldom been evaluated using a realistic visual search task. In addition, an accuracy discrepancy exists between studies that use time-on-task (TOT)—the popular method—and performance metrics for labels. Yet, to our knowledge, they have never been directly compared. Approach : This study was designed to address both issues: the use of a realistic task to elicit MF during a monotonous visual search task and the labeling type used for intra-participant fatigue estimation. Over four blocks of 15 min, participants had to identify targets on a computer while their cardiac, cerebral (EEG), and eye-movement activities were recorded. The recorded data were then fed into several physiological computing pipelines. Main results : The results show that the capability of a machine learning algorithm to detect MF depends less on the input data but rather on how MF is defined. Using TOT, very high classification accuracies are obtained (e.g. 99.3%). On the other hand, if MF is estimated based on behavioral performance, a metric with a much greater operational value, classification accuracies return to chance level (i.e. 52.2%). Significance : TOT-based MF estimation is popular, and strong classification accuracies can be achieved with a multitude of sensors. These factors contribute to the popularity of this method, but both usability and the relation to the concept of MF are neglected.
No takes yet. Share an insight, caveat, or question.
Hinss et al. (2024) studied Mental fatigue. Time-on-task (TOT) labeling vs. Behavioral performance labeling was evaluated on Classification accuracy of mental fatigue. Machine learning estimation of mental fatigue using time-on-task labeling achieved 99.3% accuracy, whereas behavioral performance labeling yielded chance-level accuracy (52.2%).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: