A Bird in the Hands: the role of verbal working memory and visuo-spatial working memory in children's receptive speech-gesture integration

The use of the gesture to complement verbal language has proven a valuable tool for learning, understanding, and remembering information. Yet, in order to fully benefit from co-speech gesturing, one must be able to successfully integrate information across the two modalities. Little is known about the sources of individual cognitive differences that may help to explain discrepancies in one's ability to integrate gesture and speech (Demir-Lira, 2018). In fact, the majority of earlier studies in the gestural literature have focused on group comparisons in order to understand multimodal language at a general level, eschewing an exploration of individual variation (Ozer and Goksun, 2020). Further, the majority of previous work has focused on the impact of childrens' gestural production on comprehension, warranting a closer look at how gestural observation benefits the listener (Wu and Coulson, 2014). The current investigation explored the relationship between two aspects of individual difference in cognition, namely verbal working memory and visuo-spatial working memory, and speech-gesture integration. Children completed tasks to gauge these two aspects of working memory and then watched and listened to a story where ambiguous speech was simultaneously clarified with gesture. Crucially, in order to understand the global meaning of the story, a child would have to successfully integrate the speaker's speech with his gesture. We found that higher rates of verbal working memory, but not visuo-spatial working memory, were associated with a greater likelihood of successful speech-gesture integration.