Brain-inspired spiking neural network frameworks for multimodal spatiotemporal brain data integration: a case study on EEG-fMRI data

Multimodal data integration advances brain informatics by capturing complementary spatial and temporal characteristics of neural activity, leading to more accurate and biologically meaningful representations of brain function. The well-established brain measurement modalities, including electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI), represent fundamental heterogeneity in their temporal and spatial feature representations, making their integration significantly challenging. This study proposes biologically inspired Spiking Neural Networks (SNN) frameworks for multimodal spatiotemporal brain data integration that incorporate early, intermediate, and late fusion strategies, including a novel explainable X-Meta classifier for late fusion. Methodologically, it extends SNN architectures to support multimodal learning across separate reservoirs for EEG and fMRI, enabling the modelling of cross-modal interactions. The fusion frameworks were evaluated on a resting-state simultaneous EEG-fMRI dataset for decoding the eyes-open and eyes-closed brain states and benchmarked against conventional machine learning and deep learning approaches. We investigated two EEG representations: raw amplitude and Power Spectral Density (PSD), each integrated with fMRI, to evaluate how the EEG feature representation influences multimodal fusion performance. Our results suggested that the proposed X-Meta classifier achieved up to 3.6% and 0.35% relative improvement in accuracy over the other fusion strategies for EOEC brain state classification, using PSD and raw EEG features, respectively. Beyond classification performance, the most discriminative features identified by the X-Meta classifier are interpretable and biologically grounded, corresponding to known functional brain regions across both EEG representations. These findings suggest that the proposed framework advances brain-inspired computational modelling by contributing not only to improving brain-state decoding but also to a better understanding of how complementary heterogeneous brain modalities contribute to brain-state representation.