Beyond the linguistic channel: an eye-tracking investigation of multimodal pun comprehension

Although traditionally viewed as mere wordplay, punning is increasingly used in cognitive research to study how the brain simultaneously activates and shifts between multiple semantic frames. While previous research has mapped the mechanisms underlying unimodal (text-based or speech-based) puns, the cognitive mechanisms driving multimodal pun comprehension remain largely underexplored. Unlike their unimodal counterparts, multimodal puns require the cognitive system to coordinate visual and verbal resources to generate humor. Using stimuli that paired short texts with images, this eye-tracking study investigated how readers process homographic and homophonic multimodal puns compared to controls. Behavioral results revealed a humor facilitation effect: Multimodal puns elicited faster reaction times and higher humor ratings than controls. Furthermore, eye-tracking data indicated a strong “text-first” modality preference governed by the length of the postpun text and the spatial layout of the text and image. Linear mixed-effects models revealed significant interactions between Condition and pun Type. Homophonic puns placed immediate, localized demands on early lexical processing within the text, whereas homographic puns shifted processing effort toward late-stage visual inspection. These findings demonstrate that multimodal pun comprehension is a dynamic, multistage process of semantic anchoring and cross-modal integration, thereby extending cognitive linguistic models of humor into the multimodal domain.