What question did this study set out to answer?

The study aims to explore how visual non-verbal cues influence speakers' acoustic adaptations during conversations.

June 17, 2026Open Access

Will I speak louder if I see you struggling to understand? Speech modifications in response to non-verbal visual cues of listening effort

Key Points

The study aims to explore how visual non-verbal cues influence speakers' acoustic adaptations during conversations.
Participants read sentences to a visible listener displaying varying levels of confusion using a head-mounted display.
Two experiments were conducted, one with background noise mentioned and one without, to isolate visual cue effects.
Participants rated their speaking effort and the listener's comprehension after each trial.
In Experiment 1, significant speech adaptations were observed across all parameters: speech rate, voice intensity, and pitch adjustments.
Experiment 2 showed persistent adaptations in most parameters despite the absence of acoustic feedback.
Adaptations occurred rapidly in response to visual cues and remained consistent throughout the trials.

Abstract

Abstract During face-to-face interactions, overt non-verbal cues from conversational partners can reflect the quality of the conversation. Here, we investigated the impact of specific visual non-verbal cues (i.e., facial expressions of confusion, approaching movements towards the speaker), on the speakers’ acoustic–phonetic adaptations typically employed to enhance clarity (i.e., speech rate, voice intensity, and pitch). Participants read sentences to a listener seated in a separate room and visible through a head-mounted display. Crucially, no auditory feedback was provided. Unknown to participants, the listener was a confederate whose behavior was pre-recorded in silence using a 360° video camera. The confederate simulated three different levels of listening effort: easy, medium, and hard. After each trial, participants rated the estimated listening effort and comprehension of the confederate, as well as their own speaking effort. In Experiment 1 ( N = 22), participants were told their voice was transmitted with varying background noise levels, providing a rationale for potential listener difficulties. In Experiment 2 ( N = 12), no mention of background noise was made, to further isolate the effects of visual cues. In both experiments, participants adjusted their speech to the seen listener’s behavior. Experiment 1 showed adaptations across all measured parameters, and significant modulations persisted for most parameters in Experiment 2. These adaptations were already present in the earliest phase of exposure to the visual effort cues and remained stable across the mini-block, suggesting a rapid and highly sensitive monitoring mechanism. These findings indicate that visible non-verbal cues of listening effort can elicit acoustic–phonetic adaptations, even in the absence of acoustic feedback or explicit communicative signals, highlighting their role in regulating communication and maintaining speech clarity.

Ask AI

Helpful

Bookmark

View Full Paper