Fatigue Detection with Spatial-Temporal Fusion Method on Covariance Manifolds of Electroencephalography

Figures

• 

  Figure 1. The framework of our model: In the SR domain, we fused the output of the SPD network and the distance information based on the Stein divergence as SR-domain features. In the TR domain: we fed the covariance matrices into a two-layer RNN with seven LSTM cells per layer, and define the output as TR-domain features. Then, the concatenated SR-domain and TR-domain features were transformed into linear features by the fully connected layer, and the features were used to give the final prediction by the softmax layer.

• 

  Figure 2. Architecture of the SPD network: The bilinear map layer consists of a transformation matrix W with full rank and its transpose. ζ of the second layer indicates a diagonal matrix, whose diagonal elements are the threshold value we set. In the logarithmic eigenvalue layer, we apply eigendecomposition to get the eigenvalues and eigenvectors of matrix Xi, which are Λi and Ui.

• 

  Figure 3. The scheme of EEG segmentation: (a) For time series of each trial, a 3 s sliding time window and a 1 s step are used to divide the EEG signals. (b) From the segmented result, the relations between different channels are captured by the covariance matrices we calculated, and we reshape them into vectors with one dimension, separately. The covariance value between the i-th channel and the j-th channel is indicated by the element C(i,j) of the covariance matrix.

• 

  Figure 4. The features of the TR domain are processed by a 2-layer RNN with 7 LSTM units per layer, where mi means the flattened matrix of the i-th segment signals, and hi indicates the i-th hidden state. The h7′ denotes the output of the RNN.

• 

  Figure 5. Simulated driving paradigm: Each lane-departure event is considered as the period from the first response offset to the second response offset, which contains the deviation onset and response onset of the current event. The time between the deviation onset and the response onset is defined as reaction time(RT), which is the main focus of our study. EEG signals were recorded during the whole task.

• 

  Figure 6. Comparison of the average RTs of 27 subjects in different driving states. The histogram with blue color (V_RT) represents the reaction time of the vigilant state, while the orange one (F_RT) represents the reaction time of the fatigue state. The horizontal dotted lines indicate the average RT of all subjects in different states. Besides, we marked the variance of reaction time, which are 0.775 and 0.043 of fatigue state and vigilant state, respectively.The average reaction time distribution of 27 subjects in the vigilant and drowsy driving state. The green histogram (V_Res) represents the response time of vigilant driving, while the purple histogram (F_Res) represents the response time of drowsy driving. The red and black horizontal dotted lines represent the average response time across all subjects in drowsy and vigilant driving, respectively. Besides, the variance of response time in drowsy and vigilant driving is 0.775 and 0.043.

• 

  Figure 7. Scatter diagram of RT values of all trials: The horizontal axis represents the value of local reaction time, and the vertical axis represents the value of global reaction time; 739 trials were labeled as vigilant states, which were marked as purple points in the lower left corner; 694 orange points in the upper right corner indicated trials with fatigue state.

• 

  Figure 8. Trajectories of variations in the inter-channel relations over time in the situation of 3, 5, 7 channel pairs. In each graph (a–f), the horizontal axis represents the 7 successive time windows, and the vertical axis indicates the relation values between different EEG channels, which are the covariance of two channels. As seen, the curve shapes within the same class are very similar to each other, whereas significant dissimilarities exist between the different classes. For example, graphs (a,d) show the dynamic variations of the same combination of 3 EEG channel pairs in MV and MD, respectively.

• 

  Figure 9. The box plots of classification accuracy of different numbers of channel pairs, which are 3, 5, 7 and all pairs. The p-values of the significance test are marked below the box plots.

• 

  Figure 10. Diagram of brain state analysis based on covariance: (a) We assumed that the electrical activity is the matrix with the dimensions of 364×2250. (b) We calculated the weight matrix set A of fatigue and vigilant states, respectively. The histograms of the third column represent the weight assignment of all channels by the same neuron in two brain states.

• 

  Figure 11. Fitted t-distribution diagrams of the weight assignment of all channels by the brain neurons, in which the blue curve denotes the fatigue state and the yellow curve denotes the vigilant state. The normalized KL divergences between two distributions are on the top of each diagram.