Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians

PURPOSE: To address the feasibility of obtaining reliable evaluations of individual physicians from peer ratings undertaken at diverse hospitals. METHOD: Eleven hospitals in diverse locations in the United States were recruited to participate. With the aid of the hospitals' medical directors, up to 40 board-certified internists with admitting privileges were recruited per hospital. Participating physicians provided demographic data about themselves and nominated physician-associates to do peer ratings. Between April 1993 and January 1994, the physicians were rated by their peers, who received a single mailing with no follow-up. The raters used a nine-point Likert scale for 11 cognitive and noncognitive categories. Administrative procedures were coordinated from the American Board of Internal Medicine. Chi-square, Student's t-test, and factor analysis using varimax rotation were used to analyze the results. RESULTS: Of the 4,139 questionnaires that were mailed to peer raters, 3,005 (73%) were returned. Of the 228 physicians who were rated, 187 received ten or more usable ratings, which were used for further analysis. The findings confirmed the results of previous research. The highest mean rating was for the category of integrity, and the lowest was for the category of psychosocial aspects of care. Ten to 11 responses per physician were necessary to achieve a generalizability coefficient of .7. Nearly 90% of the variance in the ratings was accounted for by two factors, one representing cognitive and clinical management skills and the other, humanistic qualities. For 16 physicians (9%), the ratngs of overall clinical skills were less than 7 on a scale from 1 (low) to 9 (high); their ratings for all individual cognitive and noncognitive categories were below the ratings of the other physicians. CONCLUSION: The peer raters' response rate and the analysis of the ratings suggest that the rating process is acceptable to physicians and that it is feasible to obtain reliable, multidimensional peer evaluations of individual physicians practicing in diverse clinical settings.