Regulatory Effects of Generative AI on Self-Regulated Learning and AI Literacy: A Mixed-Methods Study

This study aimed to explicitly examine cognitive patterns of prompt engineering, the role of pre-service teachers’ artificial intelligence literacy (AIL) and online self-regulated learning (SRL) abilities in their use of prompting skills to develop lesson plans. The study was designed as a convergent parallel exploratory mixed-methods study involving quantitative analysis of students’ data on achievement in lesson planning, SRL and AIL, and qualitative interpretative exploration of students’ prompt sets based on cognitive domain and argumentation levels. The quantitative data was collected from 63 sophomore college students in a Faculty of Education through a quasi-experimental study with pre and post achievement tests, an AIL scale and an online learning self-regulation scale. The participants attended the experimental procedure with AIPRO. ORG in a computer laboratory within two hours. Two raters analyzed a total of 1070 prompts on the basis of hierarchical level of taxonomy of cognitive domain, and prompt sets on the basis of argumentative characteristics in three levels. The thematic analysis of the prompts revealed that the prompts were categorized into seven different themes. Sets of statistical tests, including general linear model 2X2 ANOVA tests and t tests, were conducted with the participants’ quantitative data revealed that the study was effective in helping participants, regardless of level of SRL and AIL, to develop knowledge of lesson planning. The study then merged quantitative and qualitative results revealing implications for generative AI based college learning environments, and perspectives to inform new studies of learning and teaching with AI applications.