Lake Wobegon for Anesthesia…Where Everyone Is Above Average Except Those Who Aren’t: Variability in the Management of Simulated Intraoperative Critical Incidents

The May 2008 edition of American Vogue magazine carried the cover headline, “When Good work goes bad…undoing medical mistakes.” Inside was a story on cosmetic surgery with several tips including, “Board certification is a key indication of skill.” There was no mention of the importance of the anesthesiologist despite entertainer Joan Rivers’ claim* that the key is “to get a really good anesthesiologist …” The public perceives that board certification, or international equivalents like Fellow of the Australian and New Zealand College of Anesthetists, is the “gold standard” and, if a clinician possesses this credential, he or she has the knowledge and skills required to be competent. However, evidence is mixed as to whether successfully attaining these qualifications actually results in safe and competent practice.1 In this issue of Anesthesia & Analgesia, a research group, who have been working for many years towards a validated simulation-based test of performance ability of anesthesia providers,2,3 report that a group of board-certified anesthesiologists performed with variable, and overall mediocre, degrees of success during a mannequin-based assessment of management of simulated intra operative emergencies.4 These results are consistent with the findings of the last 20 yr of simulation-based research into the evaluation of human performance in anesthesia.3,5–11 In their study, the researchers included a comparison group of certified registered nurse anesthetists (CRNAs) who also performed with variable, and overall mediocre, degrees of success, and with only modestly (but statistically significantly) lower overall scores. Not only are the mean scores ordinary, the individual scores are not bunched-up around the average, which means that some people did very poorly. No matter how artificial the tests might be, this level of mediocre performance is unacceptable for both groups of anesthesia professionals. HISTORICAL PERFORMANCE OF ANESTHESIOLOGISTS IN SIMULATION-BASED TESTS It is now almost 20 yr since DeAnda and Gaba documented the broad variability in performance, from the inept to the expert, of management of incidents by experienced anesthesiologists, and the lack of a statistically significant difference in performance between second-year residents and experienced anesthesiologists.5 As pointed out by Henrichs et al.,4 a number of studies in the intervening years have documented the high variability of performance of experienced anesthesia personnel, and the surprisingly high rate of serious failures to respond appropriately to simulated critical events.5–8,11 We may be living in Lake Wobegon† in terms of evaluating our own abilities to manage critical events. Thus, the finding that some board-certified Anesthesiologists do not perform consistently well in simulated tests of acute care skills, and only marginally better than CRNAs, who also performed inconsistently, should not be a surprise. That does not make it any less sobering. The thought that experienced anesthesia professionals of either background, who might reasonably be expected to manage a crisis solo until help arrives in 2, 5, 10 or more minutes, may not be equipped with the skills to manage critical incidents is of great concern. WHAT ARE THE PRACTICAL IMPLICATIONS OF THIS ARTICLE? Whoever is the first line of defense when “something happens” during anesthesia should be able to demonstrate high proficiency at responding to the event. Should we not be treating this type of test in the same way we would view an advanced cardiac life support examination where we expect consistently high scores? The difference in scores between groups that would be clinically meaningful and appropriate ‘passing’ scores are difficult to estimate given the differences in clinical outcome from each of the different scenarios. For example, failure to perform some steps may lead to death in certain scenarios (e.g., no dantrolene administered for the treatment of malignant hyperthermia), versus moderate temporary morbidity in others (e.g., failure to recognize right endobronchial intubation may cause collapse and consolidation postoperatively). This is one of the limitations of using a simple checklist for which scores are the sum of key actions performed, with no weighting for important items and no egregious actions leading to failure regardless of overall score. In addition, the use of checklists tends to favor novice patterns of thinking (thoroughness versus experts who use pattern recognition). The set of skills tested include many of those generally agreed to be important for a competent anesthesia provider to possess.12 Nevertheless, in 5 of the 12 scenarios (malignant hyperthermia, blocked endotracheal tube, total spinal, hyperkalemia, acute hemorrhage), mean scores for one or both groups were <50%.4 Scenarios with ambiguous presentations (e.g., Hyperkalemia) or complex treatment protocols (malignant hyperthermia) were poorly managed. Why is it that these skills are not more uniformly and consistently demonstrated by experienced personnel? Were these skills not obtained during anesthesia training or were they gained but lost to a variable degree? Although simulation-based crisis management training has been implemented in many sites, these skills are mostly taught informally, even haphazardly, in both medical and nursing-based anesthesia-training programs. Perhaps this variability of training is reflected by the high variability in performance, ranging from poor to good with a mediocre overall average score (60%, range 38%–80% CRNAs, 67%, range 42%–87% anesthesiologists), in this study.4 Like any other skill, decay in crisis management skills will occur over time if such skills are not used or rehearsed on a regular basis. This may be a particular issue for anesthesia professionals whose practice environment is not particularly challenging. Another explanation is that there is potentially a problem with the test, i.e., it is too artificial to produce meaningful results. Both explanations may, in fact, be true to some degree. BENEFITS AND LIMITATIONS OF SIMULATION-BASED ASSESSMENT IN ANESTHESIA PRACTICE Simulation is both practical and useful in addressing issues of competence and performance in anesthesia. Competence and performance are not interchangeable terms. Miller’s model of medical competence13 implicitly assumes that competence predicts performance. However, the relationship is complicated. Rethans et al.14 extended Miller’s model of competence to develop the Cambridge Model for delineating performance and competence. Performance (what we do in clinical practice) is a product of competence (what we demonstrate under controlled conditions), systems factors (resources, organizational factors), and individual (fatigue, mood) factors. What this means essentially is that not all problems with performance are related to competence; some problems are due to organizational factors like production pressure, inadequate resources, fatigue and so on. The authors’ study tests one element of competence.4 It does not address performance. Miller’s model13 is based on the assumption that direct observation is the most valid form of assessment. Although direct workplace observation may be useful for assessing routine anesthesia, how do we assess these acute care skills in clinical practice when critical events do not occur on demand? Gaba has previously used a jigsaw puzzle analogy, with interlocking and complementary pieces, to illustrate the importance of combining data from both real and simulated situations when evaluating human performance in anesthesia.15Table 1 extends this analogy. Some facets of ability may benefit from assessment using more than one modality, e.g., nontechnical skills. In other instances, the choice will be determined on economic grounds, e.g., use of a full scale simulation center to assess perioperative planning may not be the most cost-effective use of the resource.Table 1: “Gaba’s Jigsaw”: The Importance of Using Complementary Approaches in Evaluation of Human Performance in AnesthesiaSimulation-based assessments are a useful adjunct to written and oral examinations. They are particularly useful for observing leadership and teamwork abilities and the ability to act on one’s observations, whereas oral examinations afford candidates the luxury of a subjective interpretation of future performance: “I would ask the surgeon to stop operating,” or “I would do an fiberoptic intubation to rescue the patient’s airway.” Savoldelli et al.16used mannequin-based simulation to assess 20 senior anesthesia residents and found examinees’ performance varied based on testing modality; a trainee who demonstrates “know how” in an oral examination might not be able to “show how” in the simulator and vice versa. The participant-by-modality variance was the greatest source of variance and suggested that addition of the simulator picked up strengths or weaknesses that would otherwise be missed by the oral examination. TRADE OFF BETWEEN PSYCHOMETRIC RIGOR AND THE COMPLEXITY OF THE ASSESSMENT Performance assessment in the workplace may be the gold standard, but competence assessment is logistically easier to conduct, and for some elements of anesthesia practice as mentioned above, may be the only option. Full mission simulation, which attempts to incorporate the complexity and ambiguity of real-life patient care, may more closely approximate the ideal of performance rather than competence assessment.12 Henrich et al. did not use full mission simulation in the assessment of competence; other research groups have done so.5,7,8,10,11,17 However, once the simulation begins to incorporate greater levels of complexity in order to more closely model real-life patient care, the psychometric robustness of the assessment decreases. There is a trade-off between the validity and reliability of the assessment and the complexity of the task we are measuring.12 The more we attempt to control the variables, the less closely the situation approximates real life. Conversely, the more we are able to control the variables, the more robust the test. This is the approach Henrich et al. have taken and, consequently, they have developed a very robust assessment tool. If we are to use full mission simulation for high stakes assessment, similarly robust tools will be required. However, the level of evidence required for a particular assessment should match the purpose of the assessment. If the purpose is to use the assessment as a barrier or exit examination where the stakes are clearly high, the validity and reliability of a test become critically important. If the purpose is low stakes, formative assessment, the validity and reliability become less critical. SCENARIO VALIDITY AND LEVELS OF ENGAGEMENT AND REALISM IN SIMULATION How meaningful are responses observed in a simulated environment? In other words, how representative is this observed behavior likely to be of a person’s real-life behavior? If participants are not able to say to themselves this could be real, it is difficult to expect their responses in that scenario will be meaningful, regardless of how valid a scenario might seem on other levels. There is currently a very incomplete understanding of the relationship between different aspects of realism (e.g., physical, conceptual or cognitive, emotional) and the validity of different types of simulations and scenarios for high-stakes assessment.18 Dieckmann et al.19 have recently explored these issues. The importance of understanding these issues is so that we know whether certain participants are advantaged or disadvantaged by virtue of age, background or personality features; for example, do younger people who play computer games find it easier to become immersed? KEEPING IT IN PERSPECTIVE This work has added to the work of the preceding 20 yr in laying a solid foundation for the validation of simulation-based tests and may assist in the development of courses that systematically train anesthesia providers to develop a defined set of acute care skills. The results of this study suggest that acute care skills in anesthesia should be taught systematically, reinforced throughout clinicians’ careers, and perhaps tested as a component of ongoing programs of maintenance of competency. The American Board of Anesthesiology in the United States has recently incorporated mandatory simulation-based activities into the 10-yr Maintenance of Certification in Anesthesia cycle for those with time limited (i.e., after year 2000) board certificates. If board certification and a physician’s education and training do not always guarantee quality, how do we help Ms. Rivers and others select a competent anesthesiologist? Perhaps the questions she should ask are “Are you up to date with your advanced cardiac life support and your acute care skills?” and “When was the last time you went through your paces in the simulator?” ACKNOWLEDGMENTS The author wishes to acknowledge the ideas and work of David M. Gaba (Professor of Anesthesia, Associate Dean for Immersive and Simulation-based Learning, Stanford University Medical School, Stanford, CA), and the mentorship of David Gaba and Alex Macario (Professor of Anesthesia, Health Research and Policy, Stanford University Medical School, Stanford, CA).