Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder

Reconstructing natural language from noninvasive electroencephalography (EEG) holds great promise as a language decoding technology for brain-computer interfaces (BCIs).However, EEG-based language decoding is still in its nascent stages, facing several technical issues such as: 1) Absence of a hybrid strategy that can effectively integrate cross-modality (between EEG and text) self-learning with intramodality self-reconstruction of EEG features or textual sequences; 2) Under-utilization of large language models (LLMs) to enhance EEG-based language decoding.To address above issues, we propose the Contrastive EEG-Text Masked Autoencoder (CET-MAE), a novel model that orchestrates compound selfsupervised learning across and within EEG and text through a dedicated multi-stream encoder.Furthermore, we develop a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations), which leverages pre-trained modules alongside the EEG stream from CET-MAE and further enables an LLM (specifically BART) to decode text from EEG sequences.Comprehensive experiments conducted on the popular textevoked EEG database, ZuCo, demonstrate the superiority of E2T-PTR, which outperforms the baseline framework in ROUGE-1 F1 and BLEU-4 scores by 8.34% and 32.21%, respectively.Our proposed pre-trained EEG-Text model shows the potential to improve downstream tasks involving EEG and text.This opens up promising avenues for its application in inner speech BCI paradigms, meriting further investigation.