Key points are not available for this paper at this time.
Understanding a talker’s intention conveyed through prosody is essential for successful speech communication. Although visual speech cues are typically characterized by lip movements, there are also visual gestures including head tilts and eyebrow raises that signal prosody. Listeners with cochlear implants (CIs) may rely on these visual cues to solidify their prosody perception and guard against misinterpretations, particularly because of their poor perception of voice pitch. This study used audio-visual recordings of sentences in which a talker sometimes focused one word to signify a change in meaning. After characterizing acoustic and visual prosody cues in this stimulus set, we made prosodically mismatched variants of the stimuli by transplanting prosodically focused audio onto broad focus (i.e., prosodically unfocused) video, and vice versa. Visual influences on prosody perception were measured using acoustic analysis of participant reproductions in a vocal mimicry task. Results show reproduction of F0 contour—a key acoustic marker of prosodic focus—is influenced by perception of visual prosody cues. Specifically, visual cues indicating a focused word led to higher peak F0 in vocal reproductions of that word in the sentence. These results underscore the importance of multisensory information in supporting prosody perception for listeners with hearing loss.
Fleming et al. (Fri,) studied this question.