Toward a Deep Understanding of Retinal Spike Trains via MLLMs and Diffusion Models

This is the authors' abstract. We don't add key points for this paper.

Visual information is first encoded into spike trains by retinal ganglion cells (RGCs), forming the foundation of biological vision. Decoding the semantic and perceptual content embedded in these spikes remains a key challenge in neuroscience. We introduce a novel framework that leverages recent advances in artificial intelligence to interpret spike-based visual signals. Our framework consists of spike-driven multimodal large language model (MLLM) and a spike-driven ControlNet to emulate two core functions of the visual cortex: semantic understanding and image reconstruction. By bridging biological and computational vision, our framework enables effective decoding of spike signals generated by RGCs. We further validate the model on the spike-driven MLLM on the MSCOCO test set. The experiment results demonstrate that our framework not only enhances interpretability and usability of spike-driven outputs but also holds promise as a tool to support neuroscience research and aid visually impaired individuals. Participant feedback highlights its potential to facilitate future advances in neural decoding and vision restoration technologies.