Can Better Selection Tools Help Us Achieve Our Diversity Goals in Postgraduate Medical Education? Comparing Use of USMLE Step 1 Scores and Situational Judgment Tests at 7 Surgical Residencies

PURPOSE: Use of the United States Medical Licensing Examination (USMLE) for residency selection has been criticized for its inability to predict clinical performance and potential bias against underrepresented minorities (URMs). This study explored the impact of altering traditional USMLE cutoffs and adopting more evidence-based applicant screening tools on inclusion of URMs in the surgical residency selection process. METHOD: Multimethod job analyses were conducted at 7 U.S. general surgical residency programs during the 2018-2019 application cycle to gather validity evidence for developing selection assessments. Unique situational judgment tests (SJTs) and scoring algorithms were created to assess applicant competencies and fit. Programs lowered their traditional USMLE Step 1 cutoffs and invited candidates to take their unique SJT. URM status (woman, racial/ethnic minority) of candidates who would have been considered for interview using traditional USMLE Step 1 cutoffs was compared with the candidate pool considered based on SJT performance. RESULTS: A total of 2,742 general surgery applicants were invited to take an online SJT by at least 1 of the 7 programs. Approximately 35% of applicants who were invited to take the SJT would not have met traditional USMLE Step 1 cutoffs. Comparison of USMLE-driven versus SJT-driven assessment results demonstrated statistically different percentages of URMs recommended, and including the SJT allowed an average of 8% more URMs offered an interview invitation (P < .01). CONCLUSIONS: Reliance on USMLE Step 1 as a primary screening tool precludes URMs from being considered for residency positions at higher rate than non-URMs. Developing screening tools to measure a wider array of candidate competencies can help create a more equitable surgical workforce.