Multimodal research: Addressing the complexity of multimodal environments and the challenges for CALL

Abstract Multimodality, the study of the interaction of language with other semiotic resources such as images and sound resources, has significant implications for computer assisted language learning (CALL) with regards to understanding the impact of digital environments on language teaching and learning. In this paper, we explore recent manifestations of CALL in 3-D virtual worlds, illustrated by the example of Second Life . The multimodal analyses of a conventional face-to-face lesson and three language learning activities in Second Life highlight some of the affordances and challenges presented by 3-D virtual environments. The results suggest that while multimodal resources integrate naturally to facilitate language teaching and learning in an orderly, structured and goal-orientated manner in classroom lessons, the often uncoordinated use (or absence) of avatars’ gaze, facial expression, body posture, gesture, as well as the unclear proxemics and use of space pose problems for effective communication in a 3-D virtual world. In addition, a “technology-oriented” register, alongside traditional instructional and regulative genres and registers, is introduced to help students cope with the demands of learning a language in a 3-D virtual environment. The study raises the issue of the relative effectiveness of 3-D virtual worlds for language teaching and learning. In doing so, a digital approach to multimodal research is proposed in order to address the complexity of multimodal learning environments and the various challenges for CALL.