Answers are grounded in this paper and cite their sources.
This analysis reveals the effectiveness of LLMs in generating medical QA sets, indicating significant efficiency gains.
Background NATO leaders have identified the rapid training of military and civilian first responders to deliver far-forward combat casualty care in austere, resource-limited environments during Large-Scale Combat Operations (LSCOs) as a key priority. Variability in baseline learning aptitudes complicates the achievement of this objective. Adaptive learning systems use algorithms to analyse a learner’s performance and customize educational content to meet individual learners’ needs, accommodating for differences in learning styles, and baseline aptitudes. Question-and-Answer (QA) set creation is essential to adaptive learning systems but is highly resource-intensive. Question-and-answer set creation is hindered by the limited availability of subject matter experts, high production costs, and extended delivery timelines. Objectives We describe how we leveraged generative artificial intelligence techniques to rapidly and cost-effectively generate high-quality medical QA sets consistent with National Board of Medical Examiners (NBME) standards. We analysed and described the performance of Large Language Models (LLMs ) as authors of medical QA sets and judges of QA set quality, using a human Subject Matter Expert (SME) as the gold standard. Methods A medical SME rated OpenAI model (GPT-4, GPT-4o, o1-mini, and o1) performance on 320 LLM-generated medical QA sets using a three-tier scale: ‘High’ for complete task coverage, ‘Medium’ for minor omissions, and ‘Low’ for critical omissions. Subsequently, four LLMs (GPT-4, GPT-4o, GPT-4o mini, and o1) served as judges and rated 417 QA sets using standardized judging prompts. Individual LLM performance and the interrater agreement across all four LLMs and human SME were calculated. Results When provided access to reference content, a human SME required approximately 24 minutes to author 1 high-quality medical QA set at an estimated cost of $100 USD. In contrast, each large LLM generated an individual medical QA set in roughly 20 seconds at a fraction of the cost (0.98 cents per QA set). Among the LLMs, GPT-4o emerged as the most effective author, producing high-quality medical QA sets in 86.6% of instances, medium-quality sets in 9.8%, and low-quality sets in 3.7%. The SME agreed with the high-quality ratings of QA sets by GPT-4o 87.4% of the time. The interrater agreement between the human SME and LLMs serving as judges of QA set quality was robust (Krippendorff’s alpha value of 0.85). Conclusions This study documented that LLMs can generate and evaluate high-quality medical QA sets with an estimated 31-fold reduction in time and 4,444-fold reduction in cost compared to those authored by human SMEs as illustrated in the accompanying figure 1. Large language models, when utilized as both authors and judges, with continued human oversight for quality control, can address challenges like limited availability of SMEs, as well as time and cost constraints, thereby enhancing the scalability of adaptive learning and potentially revolutionizing medical training. Future efforts involve creating customized training for first responders (e.g., appropriate use of limb tourniquets) in LSCOs by mapping QA set generation to curricular objectives within an adaptive multimodal learning system. The selected system will need to account for compromised network infrastructure (e.g., offline capability, caching, or minimal bandwidth solutions). Abstract P07 Figure 1 Time and cost advantages of LLMs in medical QA set creation Conflict of Interest (COI) Disclosures Savitsky, Eric MD – is a co-Founder, chief executive officer (CEO), and shareholder in SonoSim, Inc. SonoSim delivers a cloud-based platform dedicated to improving patient care by helping institutions and individuals achieve their ultrasound training objectives. Nataneli , Gabriele PhD – serves as the chief technology officer (CTO) of SonoSim, Inc. Lee, John PhD – N/A Koenig, Alan PhD – N/A Lindstrom, Karson – is a solutions architect at Caylent Inc., which is working as a consultant for SonoSim, Inc. Caylent combines deep Amazon Web Services (AWS) expertise and cloud-native technologies like Kubernetes, Terraform, serverless frameworks, and containerization to enable clients’ digital transformation and cloud evolution faster and with greater impact. Schriger, David MD, MPH – N/A Savitsky, Tyler – serves as a consulting computer engineer for SonoSim, Inc.
No takes yet. Share an insight, caveat, or question.
Erić et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: