AI-mediated authentic assessment and metacognitive reflection: A mixed-methods study of the AAIWA model

This study introduces the Authentic AI-Assisted Writing Assessment (AAIWA) model, integrating authentic rubric-based assessment, condition-responsive AI feedback, and structured metacognitive reflection as a unified recursive architecture within Indonesian pre-service teacher education. Existing models treat these components additively, creating a critical design gap particularly in underrepresented Global South contexts. AAIWA addresses this by computationally embedding rubric criteria within the feedback engine and positioning structured reflection as a dynamic input into subsequent feedback cycles rather than a post-hoc add-on. The AAIWA prototype is a constrained hybrid system: a rule-based NLP layer governs all pedagogically consequential decisions while GPT-3.5-turbo serves exclusively as a language-articulation module. Claims of 'condition-responsive feedback' therefore denote rule-conditioned responsiveness, not autonomous LLM reasoning. A mixed-methods R&D design engaged 75 pre-service teachers across three universities in four iterative writing cycles. Significant improvements were observed in academic writing performance (d = 0.92, 95% CI [0.65, 1.19]) and metacognitive reflection (d = 0.98, 95% CI [0.69, 1.27]). A moderate writing reflection correlation (r = .54, 95% CI [.36, .68]) offers partial mechanistic support for the integrated design. Thematic analysis (n = 15) identified three themes: enhanced metacognitive awareness, motivational benefits of immediate feedback, and progressive internalisation of reflection. Expert validation yielded strong content validity indices (rubric: CVI = 0.84; inventory: CVI = 0.82). These pilot findings associated with rather than causally attributable to AAIWA indicate a theoretically grounded, empirically promising framework warranting controlled replication across diverse educational contexts.