A Machine Learning Approach to Identifying Students at Risk for USMLE Step 1 Exam Delay Using Academic Performance Data

Abstract Purpose Delaying the United States Medical Licensing Examination (USMLE) Step 1 exam is often linked to poorer performance on Step 1, clerkship, and Step 2 Clinical Knowledge exams. This study developed and evaluated explainable machine learning (XML) models to predict Step 1 exam delay using student performance data and identify key predictors contributing to delay risk. Method Data from 610 medical students at University of Connecticut School of Medicine for matriculation years 2016, 2017, 2019, 2020, 2021, and 2022 were analyzed, with 167 students (27.4%) delaying their Step 1 exam. XML models, including RF (random forest), XGBoost (Extreme Gradient Boosting), CatBoost (Categorical Boosting), and regularized logistic regression, were evaluated using 5-fold cross-validation and testing datasets. Misclassification rate (MCR), area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC) were used to compare model performance. Shapley Additive Explanations (SHAP) and permutation feature importance were applied to interpret key predictors. Results XGBoost outperformed all other models, achieving a median (IQR) MCR of 0.14 (0.11-0.17), AUROC of 0.80 (0.76-0.84), and AUPRC of 0.72 (0.60-0.78). SHAP and permutation feature importance analyses revealed that incorrect responses on the first Comprehensive Basic Science Self-Assessment (CBSSA), along with preclerkship foundational basic science and laboratory course scores, were the most influential predictors of Step 1 exam delay. Students with delays had a median (IQR) of 99 (82-112) incorrect responses on the first CBSSA vs 75 (56-92) for nondelayers (P .001). Students with delays had significantly lower scores across the 5 preclerkship course blocks. Conclusions Ensemble machine learning models, particularly XGBoost, provide strong predictive performance for identifying students at risk for delaying the Step 1 exam. Early academic indicators, such as CBSSA and course performance, can inform timely interventions, supporting students’ academic progression and success.