Crossmodal correspondence between lexical tones and visual motions: a forced-choice mapping task on Mandarin Chinese

Abstract Crossmodal correspondence refers to the phenomenon in which individuals match stimulus features (e.g., auditory pitch) with different sensory modalities (e.g., visual size). While studies on correspondences exhibited by suprasegmentals have mostly focused on pitch-size and pitch-shape associations, audiospatial binding observed in the production and perception of Mandarin tones, where pitch of the syllable distinguishes word meanings, sheds light on the symbolic potential of auditory pitch. In the present study, a forced-choice mapping task was conducted in the form of a word guessing game, where native Mandarin listeners select the meaning of an auditory “alien” word from two visual motions. The results showed that: (1) listeners reliably match auditory tones with visual motions in the way that pitch trajectories are congruent with spatial movements, (2) vowel category impacts tone-motion correspondence when syllables are articulated in non-contour tones, and (3) the capacities in driving the tone-motion correspondence are different across tonal categories. These findings further contribute to our understanding of the sound symbolic potential of lexical tones and expand the boundary of crossmodal correspondence that can be demonstrated by pitch.