Answers are grounded in this paper and cite their sources.
The paper discusses the challenges faced by researchers in developing effective digital interfaces for analyzing the meaning-making processes of multimodal phenomena. The authors propose a social semiotic approach as the underlying theoretical foundation, because interactive digital technology is the embodiment of multimodal social semiotic communication. The paper outlines the complex issues with which researchers are confronted in designing digital interface frameworks for modeling, analyzing, and retrieving meaning from multimodal data, giving due consideration to the multiplicity of theoretical frameworks and theories which have been developed for the study of multimodal text within social semiotics, and their impact on the development of a computer-based tool for the exploration, annotation, and analysis of multimodal data.
No takes yet. Share an insight, caveat, or question.
O’Halloran et al. (2010) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: