From simulation to pedagogy: multi-model and randomized evaluation of structured AI standardized patients

Standardized patients (SPs) are central to clinical communication training but remain constrained by cost, scalability, and reliance on trained actors. We present AI standardized patients (AI-SPs), large language model–driven patient simulators built on a three-layer information architecture that adaptively modulates information disclosure according to learner skill. In Phase 1, blinded expert assessment of 350 simulated consultations generated by five frontier LLMs showed that, under a common architecture, learner skill level accounted for more of the observed performance variation than model choice ( η 2 = 0.31 vs. 0.06); whether this reflects the architecture itself cannot be determined without comparing models with and without it. In Phase 1b, analysis of 155 live student consultations identified systematic failures to elicit safety-critical information, enabling automated curriculum diagnostics without expert observation. In a Phase 2 three-arm pilot randomized study ( n = 58), OSCE performance gains with AI-SP training did not differ significantly from those with human SP training; the trial was designed to estimate effect sizes for a future confirmatory study, not to demonstrate equivalence. Self-efficacy gains were numerically largest with AI-SP training, but this exploratory difference did not survive correction for multiple comparisons. These findings suggest that architecture-driven AI-SPs could provide a scalable, model-portable approach to clinical communication training.