Cognitively Aligned Assessment Item Generation with Open-Source LLMs: A Comprehensive Evaluation on LearningQ

Automated generation of high-quality educational assessment items is still difficult, especially when it comes to higher-order cognitive skills. Although Large Language Models (LLMs) show promise, their structural validity and cognitive alignment are limited. This study systematically evaluates fine-tuning open-source LLMs using an enriched LearningQ dataset that includes Bloom’s cognitive labels and evidence. The results show a clear performance contrast. Qwen2.5-3B-Instruct displays the best semantic reasoning, while Llama-3.2-3B shows better structural adherence, achieving a 94.9% validity rate and full compliance against answer leakage while maintaining high question validity. In contrast, older encoder-decoder models like FLAN-T5-XL do not generate valid questions. The study finds that small- to medium-sized instruction-tuned models, backed by strong data engineering, are successful at developing scalable, cognitively well-aligned assessment items.