Impact of ML-LA feedback system on learners’ academic performance, engagement and behavioral patterns in online collaborative learning environments: A lag sequential analysis and Markov chain approach

Abstract Feedback is critical in providing personalized information about educational processes and supporting their performance in online collaborative learning environments. However, giving effective feedback and monitoring its effects, which is especially important in online environments, is a complex issue. Although providing feedback by analyzing online learning behaviors, it is unclear how the effectiveness of this feedback translates into online learning experiences. The current study aims to compare the behavioral patterns of online system engagement of students who receive and do not receive machine learning-based temporal learning analytics (ML-LA) feedback, to identify the differences between student groups in terms of learning performance, online engagement, and various system usage variables, and to examine the behavioral patterns change over time of students regarding online system engagement. The current study was conducted with the participation of 49 undergraduate students. The study defined three engagement levels using system usage analytics and cluster analysis. While t-test and ANCOVA were applied to pre-test and post-test scores to evaluate students’ learning performance and online engagement, lag sequential analysis was used to analyze behavioral patterns, and the Markov chain was used to examine the change of behavioral patterns over time. The group receiving ML-LA feedback showed higher behavior and cognitive engagement than the control group. In addition, the rate of completing learning tasks was higher in the experimental group. Temporal patterns of online engagement behaviors across student groups are described and compared. The results showed that both groups used all stages of the system features. However, there were some differences in the navigation rankings. The most important behavioral transitions in the experimental group are task and discussion viewing and posting, task posting updating, and group performance viewing. In the control group, the most important behavioral transitions are the relationship between viewing a discussion and making a discussion, then this is followed by the sequential relationship between viewing individual performance and viewing group performance. The results showed that students’ engagement behaviors transitioned from light to medium and intense throughout the semester, especially in the experimental group. For learning designers and researchers, this study can help develop a deep understanding of environment design.