Enhancing Neural Decoding with Large Language Models: A GPT-Based Approach

Many neural decoders specialize in one function. They provide a task-dependent interpretation of the signal based on what is happening in the subject's brain and the subject's environment when performing a particular task. We tend to improve decoder performance by simplifying paradigms and removing artifacts. However, this is far from how humans operate in nature since we justify and explain it by looking for plausible reasons and references. To build the decoder different from the conventional way and to interpret more general signals, we tried to use large language models, which are becoming increasingly popular. Like other deep learning models, large language models are highly capable of processing natural language using a Transformer. The fact that they process "language" gives them unlimited potential. OpenAI's ChatGPT, a service that uses large language models, performs various tasks. Suppose the large language model is trained to learn the characteristics of neural signals. The trained model would learn which brain region matches the Brodmann area number and the relationship between cognitive function and neural signals. In that case, a fine-tuned large language model decoder can interpret neural signals and the location where the signals occur with the learned neuroscience knowledge. This large language model-based decoder can universally interpret neural signals and give us guidelines for comprehension of dynamic brain activity. We fine-tuned the 'GPT-3.5 turbo' model and prompted it with preprocessed neural signals of each region, characterized by what bands they were in. The large language model responded with an estimate of what the neural signal was like in that state and what features it used to make this judgment. We propose that GPT can be trained with the neuroscience knowledge accumulated in the neuroscience community to create a highly reliable neural decoder.