Reliability in Evaluating Letters of Recommendation

To measure the interobserver reliability in evaluating letters of recommendation for residency applicants, three letters were collected from each of the application files of 58 residents at one program. The letters were rated by six faculty. Interobserver reliability, calculated using the kappa statistic, was slight. These preliminary results show significant variability in the interpretation of letters of recommendation.