Abstract Problem High-stakes licensing exams such as the USMLE play a critical role in medical education, influencing both trainee progression and patient outcomes. Access to high-quality board preparation resources is uneven and often cost-prohibitive, disproportionately affecting students from underrepresented or financially disadvantaged backgrounds. Approach An AI-driven system to generate USMLE-style practice questions aligned with NBME item-writing guidelines using a Large Language Model (LLM) enhanced with retrieval augmented generation (RAG), chain-of-thought and few-shot prompting, and JSON schema validation was developed and piloted at the University of Cincinnati College of Medicine between November and December 2023. Five lectures from a preclinical hematology course were selected, and 565 questions were generated for 177 first-year medical students. A human-in-the-loop process, led by a faculty course director, ensured content validity and adherence to educational standards. Validated questions were deployed via a mobile app, allowing students to practice, receive performance feedback, and access an AI tutor. Outcomes Of the 565 questions, 490 (87%) were deemed accurate and NBME-compliant. Eighty students used the question bank, completing up to 220 questions each. Although not statistically significant, increased use trended toward improved performance on related exam questions. Qualitative feedback highlighted enthusiasm for AI-assisted study tools, with calls for broader content coverage. Next Steps This pilot demonstrates that LLMs can generate high-quality, guideline-aligned practice questions. To improve scalability and reduce faculty workload, future iterations will incorporate AI-based review agents for pre-screening content. The platform is intended to be expanded to additional courses, training phases, and health professions. Ongoing refinement will focus on improving content specificity and maintaining accuracy, especially in advanced and subspecialty education.
Zahn et al. (Sun,) studied this question.