The impact of face masks on face-to-face neural tracking of speech: auditory and visual obstacles

Abstract Face masks provide fundamental protection against the transmission of respiratory viruses but hamper communication. We estimated auditory and visual obstacles generated by face masks on communication by measuring the neural tracking of face-to-face speech. To this end, we recorded the EEG while participants were exposed to naturalistic audio-visual speech, embedded in multi-talker noise, in three contexts: (i) no-mask (audio-visual information was fully available), (ii) virtual mask (occluded lips, but intact audio), and (iii) real mask (occluded lips and degraded audio). The neural tracking of lip movements and the sound envelope of speech was measured through backward modeling, that is, by reconstructing stimulus properties from neural activity. Behaviorally, face masks increased listening -phonological-errors in speech content retrieval and perceived listening difficulty. At the neural level, we observed that the occlusion of the mouth abolished lip tracking and dampened neural tracking of the speech envelope at the earliest processing stages. Degraded acoustic information due to face mask filtering altered neural tracking at later processing stages instead. Finally, a consistent link emerged between the increment of listening perceived difficulty and the drop in reconstruction performance of speech envelope when attending to a speaker wearing a face mask. Results clearly dissociated the visual and auditory impacts of face masks on face-to-face neural tracking of speech. While face masks hampered the ability to predict and integrate audio-visual speech, the auditory filter generated by face masks impacted the neural processing stages typically associated with auditory selective attention. The link between perceived difficulty and neural tracking drop provided evidence of a major impact of face masks on the metacognitive levels subtending speech processing.