A Psychometric Evaluation of AI-Generated Vs. Human-Authored Programming Questions

This study evaluates the psychometric quality of LLM-generated multiple-choice questions compared to instructor-authored items. Ten GPT-4-generated items were aligned with ten human-generated items by learning objective and administered in a summative Python programming examination to 155 undergraduates. Classical Test Theory assessed item difficulty, discrimination, distractor functioning, and reliability. AI-generated items were significantly easier and slightly less discriminating, though both sets met acceptable discrimination thresholds. AI-generated items exhibited greater quality variability and more nonfunctioning distractors, with one item showing severe psychometric failure where a distractor outperformed the correct answer. The two subtests converged closely (observed r = .62; disattenuated ≈1), indicating the same underlying construct; however, four of ten matched pairs showed non-significant correlations attributable to differences in cognitive demand or item facet rather than topic. These findings indicate that while LLMs can streamline assessment creation, their outputs require rigorous expert review and are not immediately interchangeable with human-authored items.