Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification

Figure 5. Prediction of the subjectively perceived mental effort based on a median split; validation set: N=1. Classifiers’ performance based on (A) fNIRS data, (B) physiological data (heart rate, respiration, and body temperature, (C) a combined, weighted feature set, (D) visual data, (E) performance data (accuracy and response time), (F) a combined, but unweighted feature set, and (G) a dummy classifier that considers only the distribution of the outcome classes for its prediction and represents an empirical chance level. Bootstrapped Bonferroni-corrected 95% confidence intervals (CI; 5000 iterations) of the mean F1 scores for the training set (left, orange) and the test set (right, blue) of the different unimodal and multimodal models. Notches in the boxes of the plot visualise the upper and lower boundary of the CI with the solid line representing the mean and the dashed grey line representing the median. The box comprises 50% of the distribution from the 25th to the 75th quartile. The ends of the whiskers represent the 5th and 95th quartile of the distribution. The continuous grey dashed line shows the upper boundary of the CI of the dummy classifier at 0.444.

Figure 1. Elements of the WCT interface. Left side of the screen (map): Participants had to monitor the aerial space of the airport. When an unregistered drone entered the yellow area (outer circle), participants had to warn that drone; when an unregistered drone entered the red area (inner circle), participants had to repel it. Right side of the screen (graphical user interface): Participants had to request codes and pictures of unknown flying objects and then classify them as birds, registered drones, or unregistered drones.

Figure 2. Procedure of the experiment. The presented procedure is exemplary as the task load condition was alternating, and the concurrent emotional condition was pseudo-randomised throughout the different blocks.

Figure 3. Location of fNIRS optodes. Montage of optodes on fNIRS cap on a standard 10–20 EEG system, red optodes: sources, blue optodes: detectors, green lines: long channels, dark blue circles: short channels. Setup with 41 (source–detector pairs) × 2 (wavelengths) = 82 optical channels of interest.

Figure 4. Classification procedure with cross-validated randomised grid searches (maximum number of 100 iterations) and a validation set consisting of one or two subjects. The first grid search optimises the hyperparameters for the different individual and unimodal classifiers. The second grid search optimises the weights as well as voting procedure (soft or hard) for the unimodal voting classifier. The third grid search optimises the weights as well as the voting procedure (soft or hard) for the multimodal voting classifier.

Figure 6. Weights and procedure of an unimodal voting classifier to predict subjectively perceived mental effort based on a median split; validation set: N=1. Allocated weights for an unimodal voting classifier based on (A) fNIRS data, (B) physiological data (heart rate, respiration, and body temperature), (C) visual data, and (D) performance data (accuracy and response time). Error bars represent the standard deviation.

Figure 7. Prediction of the subjectively perceived mental effort (confusion matrix of test set) based on a median split; validation set: N=1. Percentage of correctly and falsely classified perceived mental effort per model across all test subjects: TP = True Positives, TN = True Negatives, FP = False Positives, and FN = False Negatives, with “Positives” representing “High Mental Effort” and “Negatives” representing “Low Mental Effort”. For the “Best Performing Single Classifier” we selected the classifier (LDA, LR, SVM, KNN, RFC, or GNB) with the best F1 score for each subject. Confusion matrices based on (A) fNIRS data, (B) physiological data (heart rate, respiration, and body temperature), (C) a combined, weighted feature set, (D) visual data, (E) performance data (accuracy and response time), (F) a combined, but unweighted feature set, and (G) a dummy classifier representing an empirical chance level.

Figure 8. Weights and procedure of a multimodal voting classifier to predict subjectively perceived mental effort based on a median split; validation set: N=1. (A) shows the allocation of weights when all modalities are included in the multimodal classification. (B) shows the allocation of weights when performance measures are not included in the multimodal classification. Error bars represent the standard deviation.