Evaluating Competence of Anesthesiology Residents Including Procedures Performed Successfully and Entrustment Scales

See Article, page 281 This issue of Anesthesia & Analgesia features a research article by de Oliveira Filho et al1 regarding the accuracy of assessment of competence in anesthesiology residents, using the learning-curve cumulative sum method (LC-CUSUM) and its risk-adjusted form. To frame the discussion of the evaluation of competence in anesthesiology training, a brief history of the Accreditation Council for Graduate Medical Education (ACGME)'s Next Accreditation System and Milestones may be of value. The ACGME and the American Board of Medical Specialties (ABMS) introduced the 6 core competencies in 1999, thus beginning a new era of residency and fellow clinical training. These competencies were created to push forward a "shared model of professional development and [to] advance assessment of physicians in training."2 The Next Accreditation System was developed with these core competencies in mind, basing accreditation of residency and fellowship training programs on educational outcomes. To review, the goals of the system are "to enhance the ability of the peer-review system to prepare physicians for practice in the 21st century, to accelerate the ACGME's movement toward accreditation on the basis of educational outcomes, and to reduce the burden associated with the current structure and process-based approach."3 As a key part of the Next Accreditation System, the ACGME developed and introduced Milestones in 2013, which are designed to guide outcomes-based assessment of residents and fellows in the areas of the 6 core competencies (medical knowledge, patient care, interpersonal and communication skills, practice-based learning and improvement, professionalism, and systems-based practice). Each of the 6 core competency areas is further broken down into several milestones each (ie, patient care milestones individually related to preoperative evaluation and preparation, airway management, critical situation management, etc). To work from a common ground and show the science behind de Oliveira Filho's article, it is necessary to define some educational terms that are often used interchangeably but are different. The Cambridge Dictionary defines competence as the ability to do something well and competency as an important skill needed to do a job.4 Feedback and assessment are also distinct but sometimes used synonymously. Merriam-Webster defines feedback as transmitting evaluative or corrective information about an action, event, or process to the original or controlling source.5 Oxford Dictionary defines assessment as the evaluation or estimation of the nature, quality, or ability of someone or something.6 Educators make assessments to improve an individual's education. Evaluation, by design, may have limited or no practical feedback - adaptive tests reduce cost but may result in feedback to the individual that is imprecise or of uncertain applicability. An example is the board certification examination by the American Board of Anesthesiology. Each ABA examination including the In-Training Examination, and the BASIC and ADVANCED examinations (along with subspecialty certification examinations) provides a score report and keywords for questions that were answered incorrectly directly to each candidate. These keywords are designed to give a sense of the topical area for review but not the direct question(s) missed and in a time remote from the examination. As such they are designed to help for future study of topical material that may have been inadequately learned. For residency or fellowship training programs, historically, faculty evaluated anesthesiology residents on an interval basis (eg, every 6 months). Baker et al7 and Wanderer et al8 showed functionally that the approach of using detailed multidomain evaluations at such intervals would fail for many departments. Baker et al7 demonstrated the validity of this type of multidomain evaluation, generally used weekly. With a mean of 87 domain evaluations per resident, the authors used Z-scores to account for faculty leniency or severity, allowing for the averaging of the signals to achieve a reliable performance score for each trainee.7 Evaluations had 7 subscores representing the 6 ACGME core competencies; (patient care split into cognitive and technical giving a total of 7 subscores). A single factor was responsible for 82.9% of the variance in the clinical performance scores. In this way, the authors demonstrated that a Z-score can act as a single "statement" of clinical performance. Wanderer et al8 examined the psychometric reliability of Baker's Z-score system. This was accomplished by using 1200 evaluations in a balanced model for generalizability study. The authors found that there needs to be approximately 54 statistically independent evaluations to use these Z-scores for making reliable peer comparisons of the residents. There would need to be 30 independent evaluations per resident to achieve ≈90% reliability when using raw absolute scores.8 It is critically important to note that these are unequivocally high-stakes decisions for the anesthesiology resident and their patients. Considering the number of these lengthy (eg, >10-minute) evaluations required for reliability, how many residency programs nationwide would have 54 or even 30 distinct raters every 6 months completing such evaluations? These must be statistically independent ratings (eg, from 54 or 30 different faculty, or with some repeated evaluations by the same raters over sufficiently long time periods apart that absence of carryover effects is confirmed). For a resident during the second half of their CA-1 year, this may be workable, but this is impractical for many residency programs. Thus, the approaches studied by Baker et al7 and Wanderer et al8 are valid and can be reliable, yet impractical for many departments. They are also limited in not demonstrating or incorporating resident progressive improvement over time. (This does not in any way reduce the scientific value of the authors' major work; rather, we learn from their studies of the problem that the sample sizes needed are so large.) Progression or growth of anesthesiology resident performance can be evaluated in many ways. Residents complete case logs, which help detail counts of different types of anesthetics, anesthetic procedures, and subspecialty cases. The ACGME requires a resident case log to be completed against which a predetermined number of cases is required to consider whether the residency training program has sufficient count of index cases for the number of residents in the training program, not necessarily to determine if the individual resident's training is adequate (eg, 10 intrathoracic cardiac procedures using cardiopulmonary bypass). While counts help detail resident experience in certain techniques and index important cases, a simple count does not predict or evaluate competence, as we will see in de Oliveira Filho's current study. The ACGME Anesthesiology Milestones were aspirationally designed to help educators gauge resident growth and development over time against a predetermined set of competencies established to indicate progression through a training program (from no experience or have not yet met to the level of a practicing expert in anesthesiology). There are 2 ways to consider resident educational milestones: one is the process of performing a procedure, which becomes entrustable professional activities, and the second involves the number of procedures performed successfully. Introducing entrustable professional activities into the educational system focuses faculty raters and evaluators on assessing for entrustment, which "combines the evaluation of learners' knowledge, skills, and behaviors with the evaluation of their readiness to be entrusted to perform critical patient care responsibilities."9 Trainee competence is not the sole determinant of entrustment score on professional activities, unfortunately. Multiple factors have an impact on entrustment scores: trainee characteristics such as attitude and fatigue; patient factors such as case complexity and patient comorbidities; the work environment; and perhaps MOST importantly, characteristics of supervising/evaluating faculty such as leniency or severity of scoring, faculty attention to detail in the clinical work environment, and ability to interpret the entrustable professional activity assessment tool.10 Matava et al11 looked at the influence of resident and faculty gender on the performance ratings of residents within a competency-based medical education system. They used 2 workplace-based assessments: the Anesthesia Clinical Encounter Assessment and entrustable professional activities. The study revealed no gender-based differences in the entrustment of residents' assessment tools.10 Of importance, the random effects for faculty were larger than those for residents—suggesting that the variance in rater leniency/severity among faculty was greater than the variance in performance among residents.10 This can be interpreted to show that the scores for entrustable professional activities can say more about the faculty rater assigned than the resident being assessed or evaluated.10 That is the same as for when residents evaluate faculty, addressable statistically by simply adjusting assessments for knowledge of the rater's leniency/severity.12,13 In the current issue of Anesthesia & Analgesia, de Oliviera Filho et al present a study about competence instead of "simply" counting successfully completed procedures. They compare the reliability of the LC-CUSUM and its risk-adjusted form (RA-LC-CUSUM) to detect anesthesia resident competence for the performance of brachial plexus peripheral nerve blocks. As the authors detail, their study aimed to learn if these statistical methodologies can provide additional, objective data about resident competence, which may impact the clinical, high-stakes patient environment. de Oliveira Filho et al studied 32 anesthesia residents and developed individual learning curves using the LC-CUSUM and RA-LC-CUSUM. They defined competency as a maximum 15% brachial plexus block failure rate. (Remember, when contemplating their results, how poor this would be from a patient-outcome perspective.) Their results detail that 22 residents achieved competency after a median of 18.5 completed blocks in the LC-CUSUM, and 20 residents achieved competency after a median of 17.5 completed blocks in the RA-LC-CUSUM method. Critically, these techniques have a couple of key issues: the authors found substantial false positive and false negative rates with small lower limits for the 95% confidence intervals around accuracy measures, which indicate that these statistical methods may be inaccurate for high-stakes resident competency decisions. This well-designed study scientifically shows a failure of the method for practical ability to detect competence. We need to use this result! What it means is that the number of procedures completed successfully by a resident should not be the determining factor for competence. Furthermore, even with the essential adjustment for covariates such as difficulty of the block, success or failure is insufficient for reliable comparisons among residents. Rather, referring to entrustable activities, the results of de Oliveira Filho et al suggest that how the block is performed is the key to evaluating competence. We recommend trusting the science, the body of statistically rigorous work evaluating resident competence over the past 2 decades, including the current de Oliveira Filho article. While residents desire feedback regularly, that is different from evaluation. Their experience by counts and numbers helps demonstrate increasing skills and broadened clinical exposure, but simple success or failure of technical skills does not imply competence. If not done in a rigorous and validated fashion, the evaluation of residents by faculty is random and prone to the leniency/severity bias of the rater. DISCLOSURES Name: Julie L. Huffmyer, MD. Contribution: This author helped write the editorial. Name: Hallisey Estes, DO. Contribution: This author helped write the editorial. Name: Franklin Dexter, MD, PhD, FASA. Contribution: This author helped write the editorial. This manuscript was handled by: Edward C. Nemergut, MD.