Heuristics, Overconfidence, and Experience: Impact on Monitoring Depth of Neuromuscular Blockade

See Article, p Inadequate monitoring of depth of neuromuscular blockade leads to inadequate recovery of neuromuscular function during tracheal extubation and in the postanesthesia recovery unit.1 Postoperative residual neuromuscular blockade and nonuse of reversal agents after neuromuscular blockade have been associated with outcomes ranging from patient discomfort to pneumonia,2 reintubation, and even death.3,4 Yet, surveys suggest that a surprisingly high number of both the US and European anesthesiologists never use neuromuscular monitors, and most anesthesiologists do not believe depth of neuromuscular blockade monitoring should be included in minimum monitoring standards.5 Proposed explanations for this puzzling mismatch between theory and practice include evolving drugs and clinical practices, delays in translating clinical research findings to the bedside, persistence of older, less effective strategies, and inadequate knowledge.6 However, none fully explain the observed practice gap. Understanding why perioperative physicians have not incorporated routine quantitative monitoring of neuromuscular function is thus an important step toward optimizing perioperative quality and safety. In this issue of Anesthesia & Analgesia, Naguib et al7 suggest a new explanation. Via an international consortium of anesthesia societies including the International Anesthesia Research Society, Royal College of Anesthetists, European Society of Anesthesiology, Sao Paulo, Hungarian, Swiss, and Hong Kong Societies of Anesthesia, the authors surveyed 1629 anesthesiologists from 80 countries and found that anesthesiologists were overconfident in their understanding of literature regarding neuromuscular blockade pharmacology and monitoring. Naguib et al7 raise the possibility that this overconfidence may contribute to the low rate with which depth of neuromuscular blockade is monitored in clinical practice. The findings of Naguib et al7 are notable in 2 respects. First, they present objective evidence of overconfidence, a cognitive characteristic common to humans in general (and physicians in specific). An extensive literature documents the tendency of physicians to be overconfident about judgments, diagnoses, and therapeutic efficacy.8,9 Naguib et al7 add to this literature by demonstrating that in addition to the existence of overconfidence in decisions made under uncertainty (where being wrong some of the time is unavoidable), overconfidence also exists with judgments regarding well-established principles of neuromuscular blockade pharmacology. Second, Naguib et al7 raise the concerning possibility that overconfidence may meaningfully worsen clinical care. Findings by Naguib et al7 are consistent with known principles of heuristic-based behavior. Extensively categorized by cognitive psychologists, heuristics are mental shortcuts that facilitate rapid decision-making and include pattern matching (judging likelihood by similarity to previous observations rather than statistical prevalence) and anchoring (basing judgments on initial starting estimates) behavior.10 Heuristic-driven behavior is more prominent when data are not fully available, uncertainty is high, time pressure is present, or when decision stakes are high.11 Heuristic-based behavior has been previously documented in neuromuscular blockade monitoring. A 2011 study by Videira and Vieria12 observed extensive use of heuristics during decisions to reverse including inadequate breathing pattern and interval since last dose. As noted in a subsequent letter to the editor,13 such behavior raises the possibility that clinicians may prefer their own experience to published literature when making judgments related to neuromuscular blockade management. Continued practice based on perceived experience is a behavior described as an “illusion of validity” and has been attributed to 3 factors: not searching for evidence that would contradict that belief, lack of understanding of the outcomes, and the use of recollection for retrieving information.14 The added element of overconfidence suggests that self-correction is unlikely. Existing literature notes that degree of confidence is not related to diagnostic accuracy,15 and the level of confidence is often inversely related to the accuracy of the diagnosis such that physicians with the lowest accuracy in terms of self-assessment were the least skilled.16 Naguib et al7 also observe these relationships in their data. While overconfidence may play a role in not using monitors of depth of neuromuscular blockade, most clinicians believe that their patients are fully recovered from neuromuscular block when extubated.5 Because it doesn’t appear necessary, anesthesiologists are unlikely to incorporate monitoring that is challenging to use and not consistently reliable.17 Naguib et al7 clearly demonstrate overconfidence in knowledge regarding neuromuscular blocking agent pharmacology/monitoring. But their argument that such overconfidence is a potential cause of nonuse of quantitative monitoring is less clear. Barriers to adoption of routine monitoring of depth of neuromuscular blockade are numerous and include practitioner resistance to change, lack of belief that their practices and outcomes can be improved, the perceived cost of quantitative monitors and their maintenance, the time required for setup, and the relative difficulty in using them when a patient’s arms are tucked. It is thus difficult to attribute a significant portion of physician unwillingness to routinely use monitors to overconfidence. Confidence in answers has been utilized for years by The New England Journal of Medicine and the American Board of Anesthesiology through NEJM Knowledge+ (NEJM Group, Massachusetts Medical Society, Waltham, MA) and the longitudinal assessment of MOCA Minute (The American Board of Anesthesiology, Raleigh, NC), respectively. These tools use confidence ratings to both identify and address physician knowledge gaps. Both NEJM Knowledge+ and MOCA Minute present physicians with brief case scenarios that are followed by a set of response options regarding how to best treat the patient in the scenario. The scenarios require physicians to draw on their medical knowledge to determine the most appropriate treatment option. Immediately after selecting and submitting a response, physicians are asked to rate their confidence in the accuracy of their response. Only once this is done will they receive feedback as to whether their answer was correct and access to additional resources that pertain to the topic that was tested. This 2-step process—deciding on the correct response followed immediately by rating their confidence in their command of the knowledge being tested—helps physicians to identify content areas in which they would benefit from additional learning and reinforcement. The confidence ratings are used to prioritize the order in which topics are reintroduced over the course of the program. Topics in which a physician was confident but answered the question incorrectly are revisited sooner than those in which the physician was wrong but uncertain since being confidently wrong may more likely to lead to error and patient injury. Providing additional learning opportunities for topics in which physicians were uncertain of their answers is of lower priority because they are already aware of their lack of knowledge in these areas. However, the approach that Naguib et al7 describe does not clarify whether they were measuring confidence in answering the question or certainty that the answer provided was correct. Statistically, an individual who knows nothing about the topic might correctly guess the answers to 50% of true/false questions. In those instances, an individual could rightly report that they had a 50% chance of answering the question correctly although their confidence in their understanding of the material would (clearly) be lower. Accurate self-assessment requires specific training and depends on factors other than knowledge of the topic such as construction of the question and the characteristics of the tested items.18 In Dinsmore and Parkinson’s18 demonstration of confidence, mean confidence is 70% while mean correctness is only 60%—and subject matter experts, such as physicians, tend to be more overconfident. Naguib et al7 note that the confidence “gap” they observe is larger than that observed elsewhere but untrained individuals may not be well calibrated when determining their own confidence. It is impossible to know whether or not the degree of overconfidence identified by Naguib et al7 of 83.5% is different than the 70% identified by Dinsmore and Parkinson18 because survey respondents were a diverse group representing 80 countries with presumably different access to neuromuscular blocking and reversal agents, monitors of neuromuscular blockade, and teaching related to monitoring of depth of neuromuscular block. In addition, we note that the use of true/false questions may not optimally measure knowledge about the content domain being assessed. Literature support for the survey questions in the study by Naguib et al7 was, with 2 exceptions, >6 years old; 2 questions have 2 distinct parts so that each part would have to be either true or false to be answered accurately; the correct response to 1 question would depend on the timing of and the reversal agent used; and only 5 of the 9 questions reflect management of neuromuscular blockade. Left unanswered in this study is also whether the survey answers accurately reflect an individual’s practice. Without descriptive information on respondents and their clinical practice patterns, describing the response population as overconfident versus more experienced in the use of neuromuscular blockade is difficult. Finally, aspects of survey methodology may limit interpretation of the findings by Naguib et al.7 Did many respondents fail to complete the survey after accessing the material? Did respondents who completed the survey have a greater aggregate knowledge of the topic than those who did not? Without answers to these questions, the impact of findings by Naguib et al7 on clinical practice is unclear. While 1629 anesthesiologists from 80 countries completed the survey, across these many countries, differences in clinical practice and postanesthesia care unit care are likely to exist. For example, rocuronium is less frequently used in Western Australia because use of the cough suppressant pholcodine increases the risk of allergy to sugammadex.19 Additionally, training, patient populations, and clinical experience may differ widely between geographically diverse areas, and repercussions of residual neuromuscular blockade may be less severe in healthier patients. The authors also suggest that most anesthesiologists do not receive feedback on their outcomes. In an era where perioperative quality metrics such as postoperative respiratory failure (PSI-11) are tied to Medicare payment, such feedback is increasingly common. Administration of a valid Internet-based survey is challenging. Participants were only told that it was a “neuromuscular survey,” which did not anchor respondents to the task in which they were about to engage. To ensure that respondents are fully engaged, best practice is to provide a clear and complete description of the purpose of the survey and the information they will be asked to provide, as well as how they are being asked to provide their responses. The authors state that, “The aim of our study was to explore anesthesiologists’ confidence in their knowledge of the core concepts in neuromuscular monitoring. Overconfidence in this clinical domain may contribute to the persistence of inadequate detection of residual neuromuscular blockade by well-trained anesthesiologists. In the presence of overconfidence, the perceived need to confirm clinical assessments with quantitative monitoring tools is probably considered superfluous.” However, a causal link between overconfidence and neuromuscular blockade monitoring remains unproven. Attention to management of depth of neuromuscular blockade is clearly important and the problem of inadequate reversal cannot be overstated. The clinical consequences of less than optimal management, though, are less clear and need to be defined because patients frequently emerge from anesthesia with incomplete recovery of neuromuscular function.1,2,4 Should patients with a train-of-four ratio of 70% at the end of surgery be reanesthetized? Is the risk of dehiscence due to straining at the end of a procedure more problematic than deep neuromuscular blockade followed by recovery to a train-of-four ratio of only 80%? Balancing risk and gain is part of the anesthesiologist’s job. To best serve that role, the outcome information necessary to make those decisions is required—of which depth of neuromuscular block is just 1 element. Rather than attributing current clinical practice to a trait that is common in physicians, we should make it easier to adopt routine monitoring of depth of neuromuscular blockade into our clinical practices by supporting development of a quantitative monitor of neuromuscular blockade that is as easy to use as our current standard monitors, that is simple to maintain, and that provides reliable data on which we can base clinical decisions. DISCLOSURES Name: Ann Harman, PhD. Contribution: This author helped with a significant portion of the research for this manuscript and writing of the manuscript. Conflicts of Interest: A. Harman is the Chief Assessment Officer at The American Board of Anesthesiology in Raleigh, NC. Name: Avery Tung, MD, FCCM. Contribution: This author helped with a significant portion of the research for this manuscript and writing of the manuscript. Conflicts of Interest: A. Tung is a Section Editor for Anesthesia & Analgesia. Name: Chris Fox, PhD. Contribution: This author helped with a significant portion of the research for this manuscript and writing of the manuscript. Conflicts of Interest: None. Name: Cynthia A. Lien, MD. Contribution: This author helped with a significant portion of the research for this manuscript and writing of the manuscript. Conflicts of Interest: C. A. Lien has consulted for Merck in the past 24 months and served as a Director for the American Board of Anesthesiology. This manuscript was handled by: Tong J. Gan, MD.