Progress, challenges and future of linguistic neural decoding with deep learning

Language is the primary medium through which humans achieve information transfer and exchange. It enables the conveyance of ideas, concepts, and messages, thereby playing an indispensable role in social interaction and knowledge dissemination. Linguistic neural decoding aims to obtain outstanding language information from the evoked human brain during information interaction of both textual and spoken formats. In this work, we present a taxonomy of recent neural decoding progress, focusing on deep learning architectures and strategies, especially those implementing large language models (LLMs) for their powerful information understanding, processing, and generation capacity. We conclude with a concise observation of the challenges and potential future directions. This article aims to provide brain scientists and deep learning researchers with an overarching viewpoint of the significant correlations observed in the human brain during language perception and production from a methodological perspective, and thus facilitate their further investigation. This review summarizes the current progress on linguistic neural decoding from a machine learning perspective, focusing on the extraction and generation of text and speech information from brain activity patterns.