STeCANet: spatio-temporal cross attention network for brain computer interface systems using EEG-fNIRS signals

Abstract Background- Multimodal neuroimaging fusion has shown promise in enhancing brain–computer interface (BCI) performance by capturing complementary neural dynamics. However, most existing fusion frameworks inadequately model the temporal asynchrony and adaptive fusion between EEG and fNIRS, thereby limiting their ability to generalize across sessions and subjects. Objective- This work aims to develop an adaptive fusion framework that effectively aligns and integrates EEG and fNIRS representations to improve cross-session and cross-subject generalization in BCI applications. Approach- To address this, we propose STeCANet, a novel Spatiotemporal Cross-Attention Network that integrates EEG and fNIRS signals through hierarchical attention-based alignment. The model leverages fNIRS-guided spatial attention, EEG-fNIRS temporal alignment, adaptive fusion, and adversarial training to ensure robust cross-modal interaction and spatiotemporal consistency. Main results- Evaluations across three cognitive paradigms, namely motor imagery (MI), mental arithmetic (MA), and word generation (WG), demonstrate that STeCANet significantly outperforms unimodal and recent multimodal baselines under both session-independent and subject-independent settings. Ablation studies confirm the contribution of each sub-module and loss function, including the domain adaptation component, in boosting classification accuracy and robustness. Significance- These results suggest that STeCANet offers a robust and interpretable solution for next-generation BCI applications.