Variability in Standardized Letters of Evaluation: A Multi‐Institutional Review of EM Residency Based Versus Non‐Residency Based Faculty Evaluation

Background The Standardized Letter of Evaluation (SLOE) in Emergency Medicine (EM) was developed by the Council of Residency Directors in Emergency Medicine (CORD) to have a uniform approach to providing training programs with information about applicants in the match process. Recent revisions distinguish what setting letter writers originate (residency based [RB] training program vs. non‐residency based [nRB] site). Objectives The goal of this paper was to compare SLOE 2.0 scoring between letter writers from RB versus nRB settings. Methods This was a multi‐institutional cross‐sectional study. The study team from five residency programs collected data from SLOEs in their applicant pool from 2022 to 2023 match cycle. Each SLOE was reviewed for training location/SLOE type (RB vs. nRB) and numerical scores for sections A, B, and C (Anticipated Guidance [AG]). The data were not normally distributed, so were analyzed using descriptive and chi‐squared statistics. Data were examined using Spearman's Rho ( ⍴ ) to evaluate the relationship of Part A and B scores with faculty estimates of AG. Results The study analyzed 3687 eSLOEs from 1772 applicants. The majority ( N = 3526) were from RB faculty with only 161 from nRB faculty. The median scores were similar between groups, but the distribution of Part A and B scores was different between RB and nRB faculty. One exception was A4 (ability to perform common ED procedures) which had similar proportions of scores. There was a positive monotonic relationship between evaluation scores (Part A and B) and faculty estimates for AG, predicting up to 30% variability. Overall eSLOEs from nRB faculty had higher proportions of higher scores. Conclusions This study found a significant difference in proportions of scores assigned between RB faculty versus nRB faculty on most items including AG.