Addressing TikTok’s multimodal complexity: a multi-level annotation scheme for the audio-visual design of short video content

Recent years have seen a growing number of studies devoted to short video platforms, including TikTok. Due to their unprecedented popularity, the question of how meaning is produced in these videos is becoming increasingly important in terms of its social implications. However, systematic approaches to the analysis of these meanings are still scarce. In this article, we present a multi-level, multimodal annotation scheme for the genre of short videos that allows examining of both formal and functional meaning-making elements in these videos. To build this scheme on a firm theoretical and methodological basis, we first characterize the semiotic landscape of TikTok. We then proceed to explain how a combination of several approaches to the analysis of dynamic audio-visual artifacts and multimodal discourses can be useful for the study of short videos. The main part of this article presents a comprehensive annotation scheme for short videos on TikTok, focusing both on the analysis of formal audio-visual elements such as shots, visual objects, sound, and filters, as well as providing annotation categories for the description of content-based units such as events/eventualities and represented participants. We show the reliability of the scheme by testing and evaluating it on a corpus of twenty TikTok videos representing Ukrainian President Volodymyr Zelensky. We conclude the article by sketching possible future applications of the scheme in studies on larger corpora.