Facing the challenge in evaluating technology use in mobile environments

The process of developing innovative mobile approaches to informal and formal learning is challenging, not least in needing to satisfy stakeholders with diverse interests in the technology, the pedagogy and the overall system. Some approaches to evaluation may focus on examining the nature and quality of learning that occurs, while other methods may take a user‐centred approach to understand interactions with the systems. In this paper we highlight a methodology that attempts to address these two analytical issues in parallel, and to communicate the results to stakeholders. The methodology is grounded in cultural historical activity theory and is compatible with other views emerging that such evaluation can have multiple levels. The method applies task analysis to examine the conflicts that emerge when learners are interacting with technological systems in an informal learning setting. Results from a trial involving first‐aiders are used to illustrate the techniques as they were applied as part of a European project that developed a collaborative mobile learning environment. The method has been repeated in other studies and is suggested to provide a valuable tool to reflect on understanding and enable the sharing of perspectives on evaluation outcomes.