Large language models (LLMs) have emerged as promising tools for delivering scalable and personalized feedback in educational settings. This study is the first to compare an LLM (i.e., ChatGPT) with a well-established automated writing evaluation (AWE) system, Pigai, in delivering formative feedback for English as a Foreign Language learners. Using a single-blind randomized controlled design, this study involved 24 university students in a four-week academic writing training program. Participants completed weekly essays and received feedback either from ChatGPT or Pigai. ChatGPT’s responses were refined through prompt engineering, ensuring alignment with best practices for formative feedback by addressing lower-order (e.g., grammar, punctuation) and higher-order concerns (e.g., argumentation, structure), providing strengths and weaknesses, motivational language, and actionable improvement suggestions. Writing quality, perceptions of feedback quality, and motivation were assessed using linear mixed-effects modeling and independent samples t-tests. Both ChatGPT and Pigai improved writing quality over time, but only ChatGPT enhanced student motivation—particularly self-efficacy. These findings highlight the advantage of LLMs in delivering formative feedback over traditional AWE systems, particularly in fostering student motivation while achieving comparable improvements in academic outcomes. This study underscores the transformative potential of LLMs in educational technology, offering scalable and personalized feedback solutions that can address diverse educational needs.
Zhang et al. (Fri,) studied this question.