Good questions, sound measurement? Item-writing quality, psychometric functioning, and competence-level alignment of AI-generated multiple-choice assessments in chemistry education

Generative AI is increasingly used in education, yet evidence on the quality and validity of AI-generated assessments in science remains limited. This study examines whether large language models can support the development of high-quality multiple-choice tests in secondary-school chemistry. Using GPT-4, we generated an electrochemistry test from the curriculum and evaluated item quality against established item-writing criteria. The instrument was administered to 95 students at the end of an electrochemistry unit. Psychometric functioning was examined with Rasch modelling, and competence-level alignment was investigated through scale anchoring. The AI-generated items generally satisfied item-writing standards, with strengths in curricular alignment, alongside recurrent weaknesses in specialized language and option-length balance. Rasch analyses indicated acceptable fit and a stable item hierarchy, with measurement precision concentrated in the mid-ability range and distractor quality varying across items. Alignment between intended competence levels and empirical levels was weak, indicating that competence-level claims cannot be inferred from empirical difficulty alone. This study is, to our knowledge, the first in chemistry to generate a curriculum-anchored multiple-choice test with an LLM and evaluate it with mixed methods. Taken together, AI-generated items can serve as an initial draft requiring human oversight, with expert review and model-based checks before any consequential use.