Visual Components to Interpretation—Formal Approaches to Visible Meaning

Visual components of communication interact with the core linguistic system at all levels and make an important contribution to the meaning of the utterance expressed. This does not only apply to sign languages, which exclusively rely on the visual channel, but also co‐speech gestures and emojis, which are visual means that accompany and enhance spoken and written languages respectively. Visual meaning components often have iconic properties and are used to directly depict properties of the referent or event referred to. Formal semantic theories have only quite recently started to address the relevance of visual meaning components and to integrate insights from cognitive linguistics and semiotics into formal semantic and pragmatic analyses. In this review, we discuss the impact of visual components on linguistic interpretation in all three modalities (sign language, spoken language and written language). We argue that so far visual components are mainly analysed through direct applications and modest extensions of the formal linguistic vocabulary established for the semantic analysis of spoken languages; that is, depictive (iconic) meaning components are mostly analysed similarly to non‐depictive (arbitrary) descriptive meaning components. However, it is becoming more and more evident that the traditional formal linguistic repertoire developed for spoken languages needs to be extended in non‐trivial ways to meet the specific requirements of visual components and the non‐descriptive properties of visual depictions.