Generative Language Reconstruction from Brain Recordings

Abstract Language reconstruction from non-invasive brain recordings has been a long-standing challenge. Existing research has addressed this challenge with a classification setup, where a set of language candidates are pre-constructed and then matched with the representation decoded from brain recordings. Here, we propose a new method that addresses language reconstruction through auto-regressive generation, which directly uses the representation decoded from functional magnetic resonance imaging (fMRI) as the input for a large language model (LLM), removing the reliance on the accuracy of pre-constructed candidates. While an LLM can already generate high-quality content, our approach produces results more closely aligned with the visual or auditory language stimuli in response to which brain recordings are sampled, especially for content deemed “surprising” for the LLM. Furthermore, we show that the proposed approach can be used in an auto-regressive manner to reconstruct a 10-minute-long stimulus, which outperforms previous methods with a classification setup. Our findings demonstrate the effectiveness of employing brain language interfaces in a generative setup and delineate a promising path to investigating language formation in the human brain.