Association of Preexamination AI Virtual-Patient Trainer Scores With Scores on a Local Single-Station History-Taking Objective Structured Clinical Examination: Observational Cohort Study

BACKGROUND Large language model–based virtual-patient systems score history-taking practice, but whether scores correspond to independently scored examination performance remains untested. Existing evaluations emphasize internal platform metrics or simulated-patient fidelity, leaving criterion-related evidence limited. OBJECTIVE This study tests whether mean preexamination AI-trainer practice scores were associated with performance on 1 local objective structured clinical examination (OSCE) history-taking station, whether the association depended on the complaint-matched training case, and whether a model including the trainer score retained predictive performance in held-out students. METHODS At 1 medical school, second-year students received a required 6-case assignment on the AI-Powered Medical History-Taking Training and Evaluation System (AMTES) and completed an end-of-course OSCE. Each student completed 1 station rated by 1 examiner; interrater and cross-station reliability could not be estimated. The 4 OSCE diagnoses differed from the training diagnoses: 3 shared a presenting-complaint category with an AMTES case, whereas 1 OSCE case had no presenting-complaint match. Platform-to-OSCE record linkage was 474 out of 480 (98.8%); 473 students with analyzable preexamination practice formed the analytic sample. The primary predictor was the mean of normalized scores across practice records time-stamped before each student’s examination-day cut-off. Analyses used Spearman correlations, OSCE-case fixed-effects regression, and repeated 5-fold cross-validation. RESULTS OSCE scores (mean 81.5, SD 6.7) were associated with platform practice scores (Spearman ρ=0.388; P<.001). With OSCE-case fixed effects, each 1SD increase in the platform score corresponded to a 2.83-point increase in the OSCE score (95% CI 2.26-3.39). The association was observed in the AMTES-unmatched low back/flank pain stratum (Spearman ρ=0.370; n=108) and with a leave-matched-case-out score (Spearman ρ=0.398 vs 0.370 for the full mean among the same 339 students). Under a single cohort-wide cut-off at the first examination day, the association attenuated but persisted among the same 469 students (Spearman ρ decreased from 0.385 to 0.346; paired difference −0.040, 95% CI −0.065 to −0.018; regression coefficient decreased from 2.83 to 2.68 points per SD). In repeated 5-fold cross-validation (n=471), the platform-only model achieved Spearman ρ=0.374 and R2=0.176. Adding the platform score to the 2 prior physical-examination course scores increased cross-validated R2 from 0.141 to 0.236 (ΔR2=0.096; 95% CI 0.053-0.142). Adding practice volume and case breadth did not improve performance (ΔR2=−0.002). CONCLUSIONS By linking required curricular practice scores to the score on 1 independently scored local OSCE history-taking station, this observational study provides criterion-related evidence for interpreting AMTES scores. The association did not depend on including the complaint-matched training-case score. The scores also added predictive information beyond the 2 prior physical-examination course scores in internal cross-validation. These findings establish neither causal effects of practice nor within-student transfer, and residual confounding by prior ability cannot be excluded. They support further evaluation of AMTES analytics as a candidate low-stakes formative signal, with independent external validation required before operational use.