Mirror mirror on the wall, what is missing in my pedagogical goals? The Impact of an AI-Driven Feedback System on the Quality of Teacher-Created Learning Designs

Given the rising prominence of Artificial Intelligence (AI) in education, understanding its impact on teacher practices is essential. This paper presents an ABAB reversal design study conducted during a teacher training, where an AI-driven feedback system helped 19 teachers to create learning designs. It investigates the impact that the AI-driven feedback had on the quality of designs and assesses pre- and post-training shifts in teachers' perceptions of the technology. We observed statistical differences between designs crafted without (in phase A1) and with AI (B1). Notably, a small positive influence persisted even after AI withdrawal (A2). This hints that specialized AI algorithms for learning design can assist teachers in effectively achieving their design objectives. Despite noticeable shifts in teachers' perceived understanding and usefulness of AI, their trust and intention to use remained unchanged. For a successful teacher-AI partnership, future research should explore the long-term impact that AI usage can have on learning design practices and strategies to nurture trust.