A Multimodal Framework for Real-time Cognitive Load and Stress Detection Using Facial Expressions and GSR Signals

This is the authors' abstract. We don't add key points for this paper.

Stress detection in real-time requires the combination of both physiological and behavioral modalities because the use of a single modality provides limited information on the psychophysiological state of individuals. In this study, a multimodal model is proposed, wherein the physiological modality uses EDA, while the behavioral modality uses facial expression analysis. SCL and SCR features are extracted from GSR signals recorded with a sensing module developed using the ESP32 board. Simultaneously, EAR, blink rate, gaze, and head pose features are extracted from webcam video using the MediaPipe library. The features from the two modalities are time-stamped for alignment and extracted within 30 seconds' window. Then, the features are combined in the feature-level fusion process. The classification of stress levels is done using RF and XGBoost classifiers as low, medium, and high levels of stress. The results of this experiment show that multimodal fusion gives an accuracy of 91.6% compared to features from either one of the modalities.