Sensory Inputs are Encoded in Memory After Integration with Other Senses for Most Individuals

Perception of objects and events is generally multisensory, with crossmodal interactions at times leading to significant modifications in perception. However, it is not clear whether memory encoding occurs before or after these sensory integrations. Each encoding scheme—unisensory (pre-integration) and multisensory (post-integration)—has its own advantages and disadvantages. This study addresses this critical question directly for the first time. Utilizing the McGurk Illusion, where incongruent auditory and visual syllables create a fused perception distinct from individual modalities, we explored the stage at which memory encoding takes place. Across two experiments (N=121 and N=132), we found that most participants recognized the fused syllable as more familiar, indicating that sensory information is predominantly encoded after multisensory integration. These findings challenge traditional memory models that prioritize unisensory encoding, and underscore the cognitive efficiency of integrating sensory inputs to form robust memory representations.