A Deep Learning Approach for Mental Fatigue State Assessment

Figure 1. Workflow of our method. Mental fatigue stimulations (b) are applied to subjects (a). ECG data are then collected and labelled (c), followed by denoising and standardization (d), and then segmentation (e). The preprocessed data are subsequently converted into 2D images using STFT (f). Features are extracted from the 1D signal segments, 2D images, and subject-specific information (g) using ResNet and Bi-LSTM. Finally, the results are inferred through a transformer by fusing the features (h).

Figure 2. Stroop task description and three example screens.

Figure 3. Data preprocessing. The denoised ECG signal (a) is segmented into 1D time series (b) and converted into 2D spectrograms (c).

Figure 4. Feature extraction of time series data (a), STFT spectrograms (b) and physiological information (c).

Figure 5. Feature fusion using a transformer encoder.

Figure 6. Comparison between the original ECG signal and the preprocessed signal. (a) The original ECG signal data, (b) ECG signal data after denoising, and (c) the baseline-aligned ECG signal after applying the Standard Scaler.

Figure 7. The segmented ECG signal segment is passed through the STFT spectrograms. Each of the figures above displays a time series data segment with 130 points, which represents a 1-second interval.