Validity evidence for Entrustable Professional Activities assessments in surgery

Purpose Workplace-based assessments (WBAs) of surgical trainees historically demonstrate poor alignment between faculty and learner perception of clinical encounters. Given trainee perception of receiving low autonomy is associated with higher rates of burnout, depression, and attrition, it is important to better align faculty and trainee perceptions of clinical experiences. This study aims to evaluate validity evidence data, including response process, internal structure, and consequences, of vascular surgery Entrustable Professional Activities assessments (EPAs) using national pilot implementation data. Method A multi-institutional pilot implementation of vascular surgery EPAs was open to residencies and fellowships from April-June of 2024. Participating programs collected assessments of entrustment for clinical encounters. Faculty and trainee assessments were paired and descriptive statistics were conducted. A linear mixed-effects model was conducted to examine factors associated with alignment. Intraclass correlation (ICC) was calculated between trainee and faculty assessments. A random effects model was applied to demonstrate sources of variance in entrustment ratings. Linear mixed effects models were applied to examine encounter and demographic differences in faculty assessed and trainee self-assessed entrustment scores. Results Twenty-nine programs contributed 1,620 paired assessments (n = 79 trainees, n = 87 faculty). There were no differences in absolute alignment by demographics or phase of care. ICC for all EPAs combined was 0.76). Variance in entrustment ratings was attributed primarily to the interaction between EPA and trainee (31%), suggesting EPA assessments were able to discriminate proficiency within trainees across EPAs. Entrustment ratings were not different when considering demographic variables. Conclusions In this retrospective analysis of national EPA pilot implementation data of residents, fellows, and faculty, strong response process and internal structure validity evidence was established. Shortcomings of other WBAs used in surgical training, such as poor alignment between faculty and trainee perceptions of an encounter, were not evident with the use of behaviorally-anchored EPA assessments.