Presenting natural continuous speech in a multisensory immersive environment improves speech comprehension and reflects the allocation of processing resources in neural speech tracking.

Abstract Successful speech comprehension, though seemingly effortless, involves complex interactions between sensory and cognitive processing and is predominantly embedded in a multisensory context, providing acoustic and visual speech cues. Adding the perspective of ageing, the interaction becomes even more manyfold. The impact of cognitive load on speech processing has been investigated, however, characterized by a lack of realistic speech material and multimodality. In this study, we therefore investigated the effects of memory load on naturalistic immersive audiovisual speech comprehension in older adults with varying degrees of hearing impairment and cognitive capacities. By providing natural continuous multisensory speech, provided through virtual reality, we created an immersive three-dimensional visual of the speaker, and manipulated the memory load of the natural running speech inspired by a traditional n-back task. This allowed us to neural speech envelope tracking via EEG and behavioural speech comprehension in varying modalities and memory loads in a highly controllable environment, while offering a realistic conversational experience. Neural speech tracking depends on an interaction between modality and memory load, moderated by auditory working memory capacity (AWM). Under low memory load, there is an increase in neural speech tracking in the immersive modality, particularly strong for individuals with low AWM. Visually induced performance improvement is observed similarly in high and low memory load settings on a behavioural level. We argue that this dynamic reflects an allocation process of sensory- and cognitive processing resources depending on the presented sensory- and cognitive load of natural continuous speech and individual capacities.