Towards Safer Flights: A Multi-modality Fusion Technology-based Cognitive Load Recognition Framework

The cognitive overload experienced by pilots in arduous circumstances is related to the psychological state and sympathetic response of the human subject. The physiological signals of subjects are predictive and reliable in detecting their mental states, thus preventing cognitive overload. This study proposes a Multi-modality Fusion Technology (MFT) based model for recognizing pilot cognitive load through pilots’ physiological signals, including electrocardiosignal (ECG), photoplethysmography (PPG), electrodermal response (EDA), electromyography signal (EMG), respiration signal (RESP) and skin temperature signal (SKT). From these signals features are extracted and then fused at the feature layer into a united vector. This vector is then sent into the model for learning. In the decision level, individual decisions from several models are combined and a decision is finalized. Various subjects were involved in experimental data collection on a flight simulator, and the collected data were used to train and test the model. The model is evaluated through both 10-fold cross-validation and Leave-One-person-Out (LOO) cross-validation.