Key points are not available for this paper at this time.
OBJECTIVES: To evaluate the construct validity of a commercially available multimodal foundation model (Google Gemini 2.5 Pro) in assessing simulated injection laryngoplasty. METHODS: Thirty video recordings of simulated injection laryngoplasty procedures were stratified by operator experience (10 novice, 10 intermediate, and 10 expert participants). Videos were evaluated by Gemini 2.5 Pro using two prompt engineering strategies: zero-shot (rubric-based, no examples) and few-shot (rubric plus examples). Performance was compared against operator training level (ground truth). Model reliability and stability were assessed through 90 repeated inference trials. RESULTS: Under a zero-shot strategy, the model failed to discriminate between skill levels (Spearman's ρ = 0.12, p = 0.52). Conversely, few-shot prompting demonstrated a strong, positive correlation with operator experience (Spearman's ρ = 0.66, p = 0.0002) and successfully stratified skill levels (Kruskal-Wallis H = 12.4, p = 0.002). Pairwise analysis confirmed the few-shot model significantly differentiated experts from both novices (p = 0.002) and intermediates (p = 0.026). Additionally, few-shot prompting significantly improved precision, reducing mean absolute error by nearly half (0.74-0.41, p = 0.04). Reliability analysis revealed 100% ordinal consistency (75.6% exact match stability), indicating the model varied under identical conditions, but did not commit any gross classification errors. CONCLUSION: General-purpose multimodal models lack the intrinsic surgical judgment necessary to assess procedural skill. However, resource-efficient few-shot prompting successfully calibrates the model to distinguish expert from trainee performance. While promising as a scalable assessment tool, current models exhibit inherent variability that requires mitigation, such as averaging repeat model evaluations. LEVEL OF EVIDENCE: N/A.
Cheng et al. (Mon,) studied this question.