Calibrating Large Language Models for Educational Assessment: A Reliability-Weighted Contextual Calibration Framework for CTT Item Facility and Discrimination Estimation

Item facility and item discrimination are two frequently used indices in Classical Test Theory (CTT) for evaluating examination quality. This study evaluates four state-of-the-art LLMs—ChatGPT, Google Gemini, DeepSeek, and Doubao—by comparing their predicted item facility and discrimination with empirical CTT estimates computed from undergraduate mathematics examinations. To improve the consistency of LLM-generated estimates, we propose a Reliability-Weighted Contextual Calibration Framework (RCCF), which combines reliability-weighted model aggregation with contextual information from historical assessments. The framework is evaluated using a nested leave-one-item-out cross-validation (LOOCV) procedure. Experimental results show that RCCF demonstrates competitive performance across the evaluated assessment settings and CTT parameters, while its relative advantage over individual LLMs and ablation models varies across tasks. By integrating multiple LLM estimates with available historical and contextual assessment information, RCCF provides a practical framework for preliminary CTT parameter estimation, with further prospective validation needed to establish its generalizability to unseen examinations.