Hierarchical disentangled representation learning for multimodal and multi-task pilot cognitive state monitoring

During complex flight operations, multiple cognitive states may coexist and jointly influence pilot performance and flight safety. However, most existing pilot cognitive-state monitoring methods either model individual cognitive dimensions independently or fuse heterogeneous sensing modalities without explicitly distinguishing shared from modality-specific information, limiting their ability to exploit both cross-modal complementarity and cross-task common structure. To address these challenges, a Hierarchical Modality–Task Disentanglement Network (HMTD-Net) is proposed for multimodal and multi-task pilot cognitive-state monitoring. HMTD-Net performs shared–specific representation decomposition sequentially at two levels. At the modality level, modality-shared and modality-specific representations are learned under distributional similarity, orthogonality, and reconstruction constraints and subsequently integrated through multi-head self-attention. At the task level, the fused multimodal representation is further decomposed into task-shared and task-specific components. The task-shared representations form a common memory, from which each task selectively retrieves relevant shared information through cross-attention while retaining its task-specific representation. Experiments on the CLV-Pilot and MODALSIM94 datasets under both subject-dependent and subject-independent protocols show that HMTD-Net consistently outperforms representative single-task and multi-task baselines. In particular, on MODALSIM94, HMTD-Net achieves 89.52% situational-awareness recognition accuracy under the subject-dependent protocol, outperforming the strongest competing multi-task method by 2.29%; under the subject-independent protocol, it achieves 75.85% workload Macro-F1, exceeding the strongest competitor by 3.87%. Ablation studies further demonstrate consistent contributions from both levels of shared–specific decomposition and their associated constraints. These results establish HMTD-Net as an effective framework for jointly recognizing multiple pilot cognitive dimensions from heterogeneous multimodal signals.