Different reliance on sensory reinstatement and internally transformed representations during vivid retrieval of visual and auditory episodes

Episodic memory retrieval engages both sensory reinstatement and internally transformed representations. Due to modality-specific processing, auditory and visual memories may differ in their reliance on these mechanisms. We used functional magnetic resonance imaging (fMRI) and multivoxel pattern analyses (MVPA) to examine how 25 participants (12 males and 13 females) encoded and retrieved naturalistic sounds and videos. Both auditory and visual targets reinstated event-specific fine activation patterns in the association cortex during retrieval, and reinstatement strength correlates with subjective memory vividness. However, after removing encoding traces, auditory episodes showed a markedly larger reliance on internally transformed traces than visual episodes, quantified by “reinstatement-free” retrieval-retrieval similarity. Sensory reinstatement correlated more with the (detail-related) posterior hippocampus, while internal representations also correlated with the (gist-related) anterior hippocampus. Furthermore, temporal voice areas preserved gist-level (human versus non-human) information from encoding to retrieval, whereas fusiform face representations degraded. These findings reveal that auditory and visual memories share a common sensory reinstatement mechanism, but differ in the neural mechanism that supports retrieval, with participants favoring gist over perceptual details during auditory memory retrieval. Significance Statement How does the brain retrieve memories of sights and sounds, and why do their subjective qualities differ? Behavioral work suggests auditory memories may be less vivid but longer-lasting than visual memories. Using fMRI and multivoxel pattern analysis during the recall of naturalistic soundscapes and videos, we showed that auditory episodes, like visual ones, reinstated item-specific activity patterns in higher-order sensory cortex, and that reinstatement fidelity tracks subjective vividness. Critically, however, auditory retrieval relies more heavily on internally generated, gist-like representations that recruit the anterior hippocampus, whereas visual retrieval preserves richer perceptual details. These findings reveal a shared yet flexibly weighted retrieval architecture across senses, account for behavioral asymmetries in memory, and inform sensory-tailored strategies for education, ageing, and neurorehabilitation.