Using Multi-Modal Data to Cluster Student’s Behavior within Embodied Learning Context

Advances in sensor technology have supported collecting multimodal data to examine the synergistic relationship between students' interaction behavior and learning performance. However, using real-time students-generated multimodal data to analyze different students' interaction behavior during a multimodal-based embodied learning activity is under-explored. Thus, this paper explores capturing multimodal data to analyze students' interaction behaviors within an embodied learning context. We conducted an in-situ study with 40 primary school students (aged 11–12) engaging in a multimodal activity on electric circuits. We captured students' embodied interaction by collecting their eye-tracking data, emotions, gesture trials, learning performance, and time to complete the activity. Three clusters were identified using K-means clustering. Using supervised machine learning, we have identified three learning profiles based on interaction behavior features: low, medium, and high performers. The results highlighted a significant difference in the learning performance between clusters, as determined by one-way ANOVA. Our findings suggest that multimodal data representing interaction traces can encode the impact of embodied interaction behavior on learning performance.