Answers are grounded in this paper and cite their sources.
Grading physical-examination performance in objective structured clinical examinations (OSCEs) is labor-intensive, difficult to scale, and prone to inter-rater variability. In this retrospective ablation study of 300 OSCE encounters from six stations at the UT Southwestern Simulation Center, we examined how the agreement of a multimodal AI system (Gemini 2.5 Pro) with a composite reference standard changed as recording inputs were removed, and compared it with that of routine standard-of-care grading by non-physician standardized-patient evaluators (SPEs). Agreement was measured with quadratic-weighted kappa, and configurations were compared using paired learner-cluster bootstrap tests with false-discovery-rate correction. Against a composite reference standard with joint adjudication of discordant items by a physician and an emergency medical technician (EMT), three-camera native video achieved the highest agreement (quadratic-weighted κ = 0.832; 95% CI 0.775–0.885). Across isolated-modality configurations, agreement point estimates were ordered native video > audio-only > transcript-only > visual-only; the audio-only versus transcript-only difference was not statistically significant (raw p = 0.29; adjusted p = 0.36). Without audio, agreement dropped substantially and the system did not reliably distinguish performed from partially performed or omitted maneuvers (three-camera silent video κ = 0.239, 95% CI 0.167–0.307; full credit for 73.9% of items the reference scored below full credit). Agreement therefore depended largely on audio-inclusive inputs, which in OSCEs carry students’ verbal signposting and standardized-patient responses. Three-camera native video also agreed more closely with the reference than routine SPE grading (κ = 0.731; 95% CI 0.692–0.763; paired difference 0.10, 95% CI 0.04–0.16; unadjusted p = 0.003). For back range of motion, however, routine grading awarded full credit by instruction, whereas the reference applied the written rubric; without that item the difference was not statistically significant ( p = 0.17). These findings map recording requirements and limitations of AI-assisted physical-examination scoring in structured OSCEs. The locally constructed composite reference used non-physician SPE and EMT ratings; a physician and the EMT jointly adjudicated only discordant items, so the reference does not independently establish physical-examination competence. Generalization to unscripted clinical encounters remains to be tested.
No takes yet. Share an insight, caveat, or question.
Kang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: