Memory performance in hearing and vision is differentially impacted by stimulus similarity and task structure

Building unified theories of human perceptual memory requires distinguishing domain-general principles from mechanisms specific to each sensory system. To advance these theories, it is essential to compare memory performance across senses. Here, we created fair comparisons between visual and auditory memory by a) ensuring that stimulus similarity structure, the confusability of remembered items with incorrect alternatives (foils), was matched, and b) considering how stimulus presentation affects memory performance for each system. We used developments in artificial neural networks to select maximally similar and dissimilar foils for both visual and auditory stimuli. Additionally, we modified experimental parameters, predicting that presentation mode (simultaneous, sequential, etc.), and its alignment with the ecological constraints of hearing and vision, would differentially impact auditory versus visual memory. Prior work has shown a visual memory advantage, but this could be due to stimulus and task parameters. We found that maximally dissimilar auditory and visual foils are comparably confusable; however, auditory memory performance degrades more rapidly than visual memory as stimulus similarity increases, revealing a fundamental contrast in how these systems retrieve information. Adjusting stimulus presentation led to findings that auditory memory could be better, worse, or comparable to visual memory. Apparent disparities in memory capacity across vision and hearing may largely reflect differences in sensitivity to similarity structure and task. These findings establish new principles for cross-sensory memory research and demonstrate how experimental design can distort cross-sensory comparisons and undermine efforts to develop theories of memory.