Consequential Validity Evidence of Yes-No Angoff Standard Setting in a Pre-Clinical Medical School Curriculum

Abstract Background Standard setting is consequential to student outcomes for defining sufficient mastery of knowledge and thus accurately classifying medical students as competent (or not) for advancing through a pre-clinical curriculum. In particular, for multiple-choice medical knowledge exams the Yes-No Angoff method for standard setting yields consequential cut-off scores based on faculty-experts’ item-level judgments of students possessing borderline but sufficient competence required for demonstrating sufficient mastery. Adapting a construct validity framework and defining consequential academic success in terms of unimpeded progress, we investigated consequential validity evidence of passing standards derived from the Yes-No Angoff method. Methods We analyzed academic success for four pre-clinical semesters across three student cohorts. First, we identified passing standards for pre-clinical courses using the Yes-No Angoff method. Then, we applied binary logistic regression and receiver-operator characteristic (ROC) analyses with area under the curve (AUC) to evaluate passing standards. For binary outcomes, we defined academic success in terms of unimpeded progress through the curriculum and students’ first-attempt passage of the United States Medical Licensing Examination (USMLE) Step 1. Model predictors included Yes-No Angoff passing standards, Medical College Admissions test scores, and grade point averages for math and science courses. Results ROC analyses showed a low but acceptable area under the curve for a single semester in one cohort and excellent or outstanding AUCs for the remaining 11 semesters. With rates ranging between 89% and 96%, overall classification accuracy for predicting academic success was adequate for all pre-clinical semesters across all cohorts. Conclusions Affirming that Yes-No Angoff standards accurately predicted academic success, study results indicated consequential validity evidence for our school’s passing standards in a pre-clinical medical curriculum.