Visual-Guided Dual-Spatial Interaction Network for Fine-Grained Brain Semantic Decoding

This is the authors' abstract. We don't add key points for this paper.

Brain semantic decoding has received a surge of attention in the computer vision and neuroscience disciplines. However, existing techniques ignore the sparse and implicit semantic analysis issue of the brain signal, leading to coarse-grained brain semantic decoding. In this article, a visual-guided dual-spatial interaction network (VDIN) is first proposed to significantly facilitate the fine-grained brain semantic decoding, with the multiway guidance of the visual message (text and image). Specifically, the local dual-spatial interaction operation is proposed to exploit the explicit–implicit coupling semantic cues between brain and text, via the text-aware space. Moreover, the above operation simultaneously attends to the multilevel semantic cues between brain and image from the more high-resolution image-aware space. Furthermore, the global dual-spatial interaction operation is utilized to integrate and modify the obtained local semantic cues into the global one. This naturally boosts the investigation of visual–brain consistency and complementary, leading to the more expressive and fine-grained brain semantic decoding architecture. For evaluation, VDIN was verified on the public brain–visual benchmark EEGCVPR40, and the experiments demonstrated that the proposed model outperforms the best baseline with an improvement of 15.97%. Note that our model still achieves competitive performance even considering the brain-spatial space associated with fewer channels.