Evaluating the Influence of MCAT Scores on Medical Student Selection and Performance Using a Machine Learning Approach: Retrospective Cohort Study

Background The Medical College Admission Test (MCAT) has been central to medical school admissions in North America, though its necessity in holistic selection processes remains debated. The COVID-19 pandemic’s suspension of MCAT testing sessions allowed institutions to explore alternative admission criteria. Additionally, global challenges in test administration underscore the vulnerability of systems dependent on a single standardized test. Objective This study aimed to (1) assess whether including MCAT scores in admissions decisions improves the prediction of medical school performance compared with grade point average (GPA) and interview-based selection and (2) evaluate whether machine learning (ML) can generate viable MCAT score predictions when testing is unavailable. Methods We conducted a retrospective cohort study of 1898 applicants to the Lebanese American University School of Medicine (2009‐2023). Among 349 admitted students with MCAT scores, we compared admission composites including vs excluding the MCAT in relation to academic outcomes (Med 1‐4 final grades) and clinical performance (Med 1‐2 objective structured clinical examination [OSCE] scores). Students were stratified into tertiles to examine tier-specific effects. To model an MCAT alternative, we trained an ensemble ML model (least absolute shrinkage and selection operator, kernel ridge, gradient boosting, elastic net, and light gradient boosting machine [LightGBM]) using data from 1583 applicants (2009‐2020) to predict MCAT scores from cumulative GPA, core GPA, interview scores, merit points, and honors participation. The model was validated on 315 applicants admitted during the MCAT suspension (2021‐2023), and its impact on admission rankings and performance correlations was evaluated. Results Including the MCAT in the admissions composite did not meaningfully improve prediction of overall academic performance ( r =0.63, 95% CI 0.56-0.69 with MCAT vs r =0.62, 95% CI 0.55-0.68 without MCAT; P =.63). However, it significantly weakened prediction of clinical skills (OSCE: r =0.37, 95% CI 0.27-0.46 with MCAT vs r =0.48, 95% CI 0.40-0.56 without MCAT; P <.001). In tertile analyses, the MCAT modestly improved prediction among top-performing students but eliminated predictive validity in the bottom tertile ( r =0.12, 95% CI –0.07 to 0.30; P =.21 vs r =0.28, 95% CI 0.10-0.44 without MCAT; P =.003). The ML model explained 36% of MCAT variance ( R ²=0.36 with 95% CI 0.32-0.41; root mean squared error ≈10% of score range). Incorporating predicted MCAT scores reduced correlations with subsequent performance across all outcomes. Although rankings based on predicted MCAT scores strongly correlated with original rankings ( r =0.80, 95% CI 0.77-0.83), 6.3% (4/64) to 9.4% (6/64) of admission decisions would have changed. Conclusions Within this single-institution study, MCAT scores provided limited incremental validity for predicting academic performance and reduced prediction of clinical skills. ML-based predictions of MCAT scores introduced sufficient error to affect admissions decisions, supporting a robust holistic admissions process during temporary MCAT disruptions while highlighting the need for validation across diverse institutional settings before broader generalization.