Abstract Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results (r = 0. 988) using only 1. 3% of items. Evaluation time decreased from 6. 85 hours to 8. 4 min per model, and token usage dropped from 1. 77 million to 0. 03 million. Model rankings were fully preserved (Spearman’s ρ = 1. 0). At current API pricing, per-model evaluation costs fell from approximately 1, 475 to under 5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.
Zheng et al. (Mon,) studied this question.