AI-assisted MCQ creation increases item-writing flaws through automation bias: evaluation of a workflow of multiple AI agents with teachers

Introduction Artificial intelligence (AI) holds promising potential for generating educational resources such as personalized assessments. Despite this appeal, the impact of AI-assisted multiple-choice question (MCQ) authoring on item quality and on teachers' editing behavior remains insufficiently studied. Methods This study (N = 152 items from 19 teachers) compares MCQ quality across three conditions—teacher-only (T), AI-only (AI), and teacher–AI collaboration (T-AI)—using a 19-criterion item-writing-flaw (IWF) rubric and logged authoring-interaction density. Results Neither human raters nor AI models reliably differentiated item provenance, indicating that LLM-generated MCQs have reached a level of surface quality largely indistinguishable from human-authored material. Yet both AI and T-AI items carried significantly more flaws than T items, with the T-AI condition showing a marked increase ( d = 1.06) consistent with an automation-bias pattern: teachers accepted AI drafts with minimal critical engagement, as evidenced by significantly lower interaction density in the AI-supported condition. Criterion-level analysis revealed a complementary pattern: AI items exhibited specific cueing biases—notably lexical overlap between stem and correct answer—but simultaneously avoided structural format flaws (e.g., true/false questions) that teachers commonly produced. Discussion These results underscore that the human–AI interaction workflow, rather than AI capability per se, is the critical quality determinant, and that AI-assisted assessment creation requires structured quality review processes that leverage AI's structural consistency while correcting its cueing biases.