Development and Evaluation of an LLM-based Automated Feedback System for Reflective Writing in K-12 Education

This study addresses the challenge that severe time constraints in Japanese K-12 classrooms prevent teachers from providing continuous, individualized feedback on students’ reflective writing. To bridge this gap while ensuring pedagogical validity, we developed a real-time generative AI feedback system grounded in the Intelligent Tutoring Systems (ITS) architecture. The system explicitly integrates three ITS components: (1) a Domain Model encoding curriculum-standard-aligned learning objectives and criterion-referenced assessment rubrics; (2) a Pedagogical Model that operationalizes Hattie and Timperley’s four feedback levels (task, process, self-regulation, and self) through structured prompt design, incorporating few-shot exemplars co-authored by veteran teachers; and (3) a Learner Model synthesizing survey-based student profiles (self-efficacy, self-regulation, and reflection tendencies), long-term reflection-pattern summaries, and recent reflection–feedback sequences. A defining design feature is the simultaneous presentation of two AI personas: an Affective Support Persona that acknowledges learners' efforts and provides warm encouragement, and a Metacognitive Deepening Persona that poses reflective questions and offers perspectives for subsequent learning. To minimize expectancy effects, persona labels were hidden and display positions were randomized. The system was deployed during routine lessons across six classes at two elementary schools and one junior high school (N = 100; November–December 2025). Acceptability was assessed using a three-dimensional scale—clarity, specificity, and empathy— each measured by a single exploratory item grounded in the framework of Lizzio and Wilson. Across 397 lesson-level evaluations and teacher surveys, both personas were highly accepted (all means above 4.1 on a 5-point scale) by students and teachers alike (RQ1). For RQ2, students rated the Affective Support Persona higher on empathy, whereas teachers rated the Metacognitive Deepening Persona higher on both specificity and empathy with medium effect sizes, reflecting an emphasis on actionable guidance that advances learning. Although the middle-achieving group showed a significant persona difference in empathy, Kruskal-Wallis tests detected no between-group differences across achievement levels, suggesting that observed patterns primarily reflect differences in statistical power. Qualitative teacher responses highlighted the system’s potential to reduce workload while raising concerns about hallucination, student over-reliance on AI, and possible weakening of teacher-student rapport. Rather than replacing teachers, the system automates immediate, theory-grounded feedback to complement instruction, allowing teachers to reallocate effort toward context-sensitive coaching. This work provides a design framework integrating feedback theory and ITS architecture for LLM-based feedback in K-12 education, demonstrates the novelty of structured dual-persona presentation, and offers empirical evidence of acceptability from both student and teacher perspectives through sustained classroom deployment.