Can Multimodal Large Language Models Annotate Low- and High-Level Features of Multimodal Artefacts?

Multimodal large language models (MLLMs) have sparked interest among scholars in digital humanities as they present a viable method for large-scale analysis of multimodal data. For the study of multimodal communication, one potential application is corpus annotation, addressing the challenge of creating richly annotated corpora which demands significant time and expertise. This study explored GPT-4o, a state-of-the-art MLLM, for two foundational annotation tasks in building multimodal corpora of primary school science diagrams: (1) identifying individual modes of expression, such as written language, illustrations and arrows, and detecting their locations in the diagram layout; and (2) classifying the type of each diagram. Our quantitative evaluation against 1,000 human-annotated diagrams revealed that, while GPT-4o can identify the presence of expression, it cannot detect their location in the diagram layout. In classifying entire diagrams, GPT-4o can classify certain diagrams with moderate accuracy but struggles with others. Our results show that MLLMs cannot be used for annotating multimodal data as an out-of-the-box solution, due to their inability to perceive the structure of multimodal artefacts through pixel representations.