Advancing Item Writing Methods with Large Language Models and Subject Matter Experts

This paper examines the use of generative large language models (LLM) to produce selected-response (SR) test items. In study 1, we compared 201 prompt-engineered LLM-generated items to 52 items developed by subject matter experts (SME). We found that 60% of LLM-generated items met psychometric criteria, compared to 74% of SME items. LLM-generated items had an average difficulty (0.73) and discrimination (0.16) similar to SME items (0.78 and 0.18). However, LLM-generated items exhibited a higher frequency of psychometric flags (e.g., high difficulty) and higher initial rejection rates. In study 2, a fine-tuned LLM generated 154 SR items across two batches. Fine-tuning improved item retention rates by 16% when compared to prompt engineering and by 10% between batches. Batch performance showed stable difficulty (0.69) and improved discrimination (0.24 to .27). While LLM cannot yet replace SME-driven item development, it may offer efficiency gains when integrated into the item development model.