Measurement of the weight of academic performance on the residency interview and ranking

Background:Educators agree that no one factor predicts a resident's chance of success in residency and beyond, and academic record is likely the strongest biasing factor in residency selection, though the exact magnitude of its weight is unknown.Objective: To determine, with respect to medical students applying to residency in emergency medicine, to what degree prior knowledge of the candidate's academic record affects their ranking.Methods: We undertook a prospective observational study, analyzing ranking scores of all interviewers of applicants to our East Coast academic EM residency program in the 2022-2023 interview cycle.Each applicant underwent 4 separate interviews, but (randomly) one of the three interviewers was blinded to their academic record.Though the applicants were interviewed, it was the interviewers who were the true subjects, and IRB approval was obtained.Applicants were then scored on a ten-point scale and data was stored in a secure database.These scores were then analyzed for inter-observer agreement.A difference of an interview rating score of 10% or greater was considered significant.Data was analyzed using a Student's T-test and Mann-Whitney test to compare data.Results: 176 interviews were included for analysis.Interview scores between blinded and unblinded interviewers were significant (p<0.00001).When the differences were spread out via histogram, the discordances were significant at 1.5 points (p<0.0001) and statistically significantly related to percentile scoring on USMLE or COMLEX.The candidates with the highest blind/unblind discordances were associated with more failures in medical school (p<0.03)and on standardized exams like the USMLE or COMLEX (p<0.07) for poor academic performers.Conclusions: This data would suggest academic performance accounts for at minimum a 15% jump or drop in rank score when assessing final applicant rank.Expanding this type of study may give insight into both interview biases.