Pilot visual cognition modeling: A cross-attention fusion model based on visual gaze behavior and neural signal-assisted multimodal data

Constructing a computational model that accurately depicts the “visual-cognitive” process of pilots is a crucial step in enhancing the efficiency of aviation intelligent interaction systems. This paper proposes a pilot flight information cognitive analysis method based on multimodal data. This method is grounded in the theory of situational awareness and implements implicit cognitive detection in simulated flight experiments, forming a closed-loop cycle of “task issuance - information acquisition - operation execution”. Through post-experimental data analysis, the “task issuance” and “operation execution” stages become observable, enabling backward inference on the “information acquisition” process and obtaining high-confidence data labels. To overcome the limitations of traditional visual gaze analysis methods, this paper introduces electroencephalogram (EEG) signals based on the multi-feature analysis of visual gaze with a fusion of long and short time windows, achieving the cross-fusion of behavioral and neural data. Furthermore, a Cross-Attention Fused LSTM (CAFLSTM) model is constructed to address the problem of associative modeling among multimodal features using a cross-attention mechanism. The experimental results show that the model can effectively identify the pilots' cognitive state of different flight information, and the accuracy rate is 94%. The proposed method provides a feasible technical path for developing intelligent pilot state monitoring and adaptive interaction systems.