Quality Assurance Data for Residentsʼ Global Performance Ratings

The Accreditation Council for Graduate Medical Education (ACGME) has defined six areas of competency that are expected of a new practitioner. Residency programs must require their residents to master competencies in the six areas, but little guidance exists about how to assess these competencies. To assist residency programs, the ACGME has developed the Toolbox of Assessment Methods.1 The Toolbox recommends global ratings as a potentially applicable assessment method for selected competencies in two areas (patient care and practice-based learning and improvement). Global performance ratings are the most widely used measure of residents' performances. A literature review of global ratings in residency education concluded that they provide relatively reliable and consistent evaluations.2 When used in an OSCE, global ratings have been shown to be reliable and valid.3 Global ratings used over time in actual clinical settings, however, have been described as “… little more than a popularity contest…” due to the unreliability of raters' memories and the paucity of observation.3 Other problems with global performance ratings in actual clinics (GPRACs) are leniency error, which also contributes to the low numerical precision of individual ratings,4 and the lack of behaviorally specific written comments.5 Many criticisms of the GPRAC as a measurement method would be less serious if the GPRAC were viewed as a low-cost, noninvasive epidemiologic screening test for marginal performance, analogous to a blood pressure screening. Prescriptive screening is aimed at detecting disease that can be better managed when detected early in presumptively healthy individuals.7 Continuing this screening analogy, a substandard (below-average) GPRAC for John Q Resident (i.e., positive screening test) would indicate possible marginal performance. However, in order to diagnose and treat the problem, additional data must be obtained, such as assigning John Q to a rotation with intensive observation by several attending faculty. If GPRAC were viewed as a prescriptive screening test, then measurement problems such as low numerical precision of a single observation and paucity of observation would be less serious because additional diagnostic performance data would precede the initiation of definitive action. This paper proposes a method to generate quality assurance data for the GPRAC. The completed rating forms are only one of four parts (context surrounding the ratings, faculty ability and motivation to judge performance, completed rating forms, and the program's willingness to take administrative action regarding a marginal resident) that comprise the GPRAC system.5 The four parts of a GPRAC system function interdependently as a unified whole. A GPRAC system can be improved by making changes to one or more of the system's parts. Quality assurance addresses the question of how GPRAC quality can be ensured and managed. Internal quality monitoring provides quantitative data to assess how well a system is functioning.6 This study proposes an internal quality monitoring system for the GPRAC based on four prerequisite conditions that should exist in a residency program. First, GPRAC forms must be routinely completed and returned to the residency program director. Second, attending faculty must write candid appraisals when a resident's performance is marginal. (A national survey of residency directors indicated that about 7% of residents have problems in one or more areas of competency.8) Third, the program must be willing to act by diagnosing and by treating a performance problem that is identified by a small number of GPRACs. Fourth, the program should have a systematic GPRAC database to support more severe administrative actions when educational diagnosis and treatment fail. Internal data for quality monitoring can be collected for each prerequisite condition described above, specifically: What percentage of all rotations complete GPRAC forms and return them to the program director? What percentage of forms communicate possible performance problems? What is the probability that the program will take administrative action when performance problems are communicated? How reproducible are the numerical and narrative performance data supporting severe administrative actions? This study provides operational definitions for the four internal quality monitoring questions and presents data from a residency program to demonstrate how answers to these questions point to needed improvements in a GPRAC system. Methods Data for this study were all global ratings and associated written narrative comments generated during the academic year 1998–99 from an integrated general surgery residency program. We excluded data on residents who were not available for at least two consecutive years beginning in 1998. Thus, no preliminary first-year or categorical fifth-year (chief) residents were included in this study. The rating form used to evaluate all residents consisted of six performance categories (fund of knowledge, clinical abilities, technical abilities, interpersonal relations, personal qualities, and teaching ability) plus an overall global rating on a seven-point scale (needs improvement: 1–2, satisfactory: 3–5, excellent: 6–7). Rating forms were sent before the end of a rotation to all faculty and the chief resident on the service. Raters were asked to compare each ratee with fellow residents at the same level of training. Narrative comments were requested for performance categories rated above or below satisfactory. If the global rating was 1, 2, or 3, the rater was asked to recommend an action (letter of concern, warning, probation, leave of absence, suspension, or termination). Data analyses were conducted to answer four internal quality monitoring questions: What percentage of all rotations completed GPRAC forms and returned them to the program director? All ratings available for a given resident were reviewed against the number of rotations completed by the resident. If at least one rating form existed for each rotation, the resident's file was scored as 100% complete. Thus, return rates could vary from 0% to 100% for each resident. To answer the question, a mean return rate was computed across all residents in the program. What percentage of GPRAC forms communicated possible performance problems? All rating forms returned were reviewed to identify possible performance problems communicated by marginal numerical scores (global ratings of 1, 2, or 3) or marginal written comments (e.g., a concern or deficiency expressed by the evaluator). To answer the question, the total number of forms communicating possible performance problems was divided by the total number of forms returned. What was the probability that the program would take administrative action when possible performance problems were communicated? For each form indicating marginal performance (i.e., below satisfactory appraisal), the residency program's records were reviewed to determine whether any administrative action had occurred (e.g., letter of concern, warning, probation). A 2 × 2 matrix was constructed to calculate sensitivity, specificity, and predictive power of a marginal performance rating. The predictive power of a marginal performance rating was defined as the probability that the program would take administrative action. To answer the question, the total number of administrative actions was divided by the total number of forms identifying marginal performance. How reproducible were the numerical and narrative performance data supporting severe administrative actions? A standard error of measurement (SEM) was calculated for the numerical scores to provide a 95% confidence interval around mean rating scores for individual residents. The SEM used a generalizability coefficient based on ratings by all evaluators for each resident. The narrative comments for each substandard numerical rating were reviewed to determine whether specific performance problems were defined. Results A total of 474 rating forms for 35 categorical residents (postgraduate years of training 1–4) were available for analysis. The average percentage of rotations that returned forms across the 35 residents was 72.6%. Seventeen forms had numerical ratings that suggested problems (global ratings of 1, 2, or 3), and 13 forms had written comments that suggested problems. Some forms had both marginal ratings and marginal written comments. Thus, 30 of the 474 forms (6.3%) indicated possible performance problems. These 30 forms identified 13 residents, who each received from one to five marginal appraisals (ratings, comments, or both). A review of the program's records regarding the 30 forms demonstrated that 13 forms (ten residents) resulted in administrative action (the data are summarized in Table 1). The sensitivity is 1.0, the specificity is .96, and the predictive value of a marginal rating form is .43 (43.3% probability that the program would take administrative action). Generalizability analysis of the 474 forms produced a coefficient of .59 based on ten raters (the numbers of raters ranged from three to 25, with a median of 13). The 95% confidence interval for a resident's mean rating based on ten raters was ±1.16 on a scale that ranged from 1 to 7.TABLE 1: Numbers of Individual Resident Global Rating Forms That Recommended Acceptable or Marginal Performances and the Related Actions by the Resident Education Committee, Department of Surgery, University of Texas Medical School at San Antonio, 1998–99Discussion The percentage of rotations that completed rating forms (72.6%) was low. In a national survey of internal medicine residency directors, 66% estimated their return rates to be higher than 75%.7 Rotation return rates for individual residents ranged from 33% to 100%. In general, forms were more likely to be returned for satisfactory and excellent residents, and written narrative comments were more likely when the global rating was 4 or lower. All of the forms with global ratings of 1, 2, or 3 had written comments describing specific performance problems. Although the administrative task of collecting and processing numerous forms is challenging, the return rate increased to 97.4% in the academic year 1999–00. We believe that the improved return rate was due to the hiring of an educator who assumed responsibility for the GPRAC system, provided feedback to faculty about administrative actions and their timely completion of forms, and shared GPRAC information with residents. Without a relatively high return rate (e.g., 90%), it is difficult to be confident about administrative decisions due to missing performance-appraisal data. Other investigators have identified a low percentage of forms that communicated performance problems as a problem in itself as we did in this study (6.3%). For example, it is difficult to convince a marginal resident of a potential deficiency if almost all of the resident's appraisals are positive.8 In this study, when a form communicating a potential performance problem was received, the program was quite likely to take administrative action, which varied from counseling to probation (probability = 43.3%). Thirteen of the total of 35 residents received at least one marginal appraisal (numerical ratings, comments, or both). Of these 13 residents, administrative action was taken with respect to ten. When making administrative decisions, the Residency Education Committee considered a wide variety of information in addition to the ratings and comments. For example, an action might not be taken if there were no data from other services to support the proposed action. For this study, we reviewed official documents that substantiated the administrative decisions. Viewing the GPRAC as a prescriptive screening test for identifying marginal performance allows for the calculation of measures such as sensitivity and specificity by considering that the department's administrative action is analogous to determining true status of an underlying performance problem. In this study, sensitivity was 1.00 because the department never took administrative action without negative appraisals, and specificity was .96. The predictive value of a negative appraisal was .43 (i.e., the probability that the department would take administrative action). It is important for a residency program to communicate the predictive value of a negative performance appraisal to its faculty. If faculty suspect a low probability of administrative action, they may decide not to write candid appraisals of marginal residents in order to avoid interpersonal conflicts with residents or other evaluators. This phenomenon could contribute to a positive bias in global ratings that has been called the mum effect.9 The generalizability coefficients for one and ten raters in this study (.13 and .59, respectively) were similar to those reported by others.4 These coefficients suggested that the 95% confidence interval of numerical scores for a resident based on one rater were not very precise (±.77), but that the 95% confidence interval of numerical scores based on ten raters was somewhat more precise (±.53). In addition, the program had written narrative comments for all of the marginal numerical ratings to further document substandard performance. The only “gold standard” of satisfactory clinical performance is the collective judgment of attending faculty who work with a resident. There are at least three limitations to this study. First, it is difficult operationally to define a written narrative comment that communicates a possible performance problem. Two researchers (JL and JP) read all of the comments and identified phrases or sentences that conveyed potential problems about a resident's performance. Previous researchers have suggested that it is helpful to be familiar with individual clinical faculty who write performance appraisals to appropriately interpret their comments.10 Some narrative comments in this study had additional meaning to the director of surgery education (JP) because she interpreted them in the context of the comment writers' norms and the specific rotations. To replicate these results, the interpretation of written comments would be aided by using an analogous “informed” reader. Second, defining the program's administrative actions as analogous to determining the true status of an underlying performance problem has an element of circuitous logic. The predictive value of a negative appraisal is directly related to the program's willingness to take administrative action. If the program is very hesitant to take administrative action, then, from a screening test perspective, faculty who write negative appraisals are likely to make false-positive errors. As noted above, the only “gold standard” regarding satisfactory clinical performance is the collective judgment of the faculty who work with a resident. If a program takes no administrative actions with residents, then, by definition, individual faculty who write negative appraisals have made errors. Finally, residents receive multiple performance appraisals just as patients receive multiple prescriptive screening tests. Individual appraisals served as the unit of analysis, but they must be interpreted in the context of the rotation and other available data (e.g., the 13 true positives included only ten residents). We believe that these limitations do not compromise the appropriateness of prescriptive screening as a quality assurance measure for the GPRAC. As noted earlier, this study describes an approach for internal quality measurement of the GPRAC process.6 Data such as those gathered in this study are best used by a residency program's education committee for self-assessment of its performance assessment system for residents. System improvements can be developed, such as recognizing that competence assessment depends on several concurrent pieces of evidence, including careful observation of actual clinical performance.11 In addition, these quality assurance data are useful for external quality assessment by groups such as residency review committees.