• EEG was recorded as listeners watched congruent and incongruent speech-gesture clips. • EEG was reconstructed from the speech envelope via temporal response functions (TRF). • Gesture congruency modulated TRF to speech during the interval of the auditory P2. • EEG during congruent videos also yielded superior decoding of the speech envelope. • Congruent gestures increased the fidelity of the brain’s encoding of speech prosody. Here we investigate how co-speech gesture kinematics impact speech tracking – a neural index that captures the brain’s sensitivity to acoustic parameters of continuous speech, especially low-frequency fluctuations in the amplitude of the speech envelope. To do this, we recorded EEG from 36 healthy adults as they watched a series of multimodal discourse clips, wherein the speaker’s movements and gestures were either congruent or incongruent with their speech. Next, we constructed temporal response functions (TRFs) to decode speech amplitude information from EEG recorded as participants watched the discourse clips. Our analyses revealed a significant reduction in decoder performance for models trained on speech presented with incongruent gestures relative to congruent ones, suggesting the fidelity of the neural representation of the speech envelope was greater for speech accompanied by congruent gestures. Next, because the spatial and temporal properties of the forward TRF can be interpreted in a similar way to traditional event-related potentials, we constructed forward models of EEG elicited by the speech envelope. These analyses revealed that weights from the forward models began to diverge across conditions 200 ms after changes to the speech envelope as TRFs to speech paired with congruent gestures were suppressed relative to TRFs from incongruent speech-gesture pairings. Results from the current study indicate that bodily gestures influence cortical tracking of the acoustic envelope of continuous speech. These gesture congruity effects may result because expressive movements of the body influence the amplitude and pitch of accompanying speech, and the brain actively exploits their informativity for speech prosody.
Momsen et al. (Wed,) studied this question.