Do As We Say, Not As You Do: Using Simulation to Investigate Clinical Behavior in Action

When mannequin-based simulation was first introduced into regular teaching of clinicians in the late 1980s and early 1990s, a common perception was that the technology would be good only for early learners, for learning procedures, and was—at best—a “toy.” Over the years, the applications of the technology have multiplied greatly. The application of simulation to “nontechnical” or behavioral skills has become commonplace. This has been extended substantially to also cover arenas of higher level decision-making concerning issues of professionalism and indeed of ethics, and these issues are dealt using simulation in many sites, in many disciplines and domains of healthcare. For example, my simulation colleagues in Emergency Medicine in 2004 described1 the embedding of “ethical probes” into nearly all their simulation scenarios in their Emergency Medicine CRM courses. Although most simulations are still targeted at education and training, the use of simulation for research on patient care processes and clinician performance has also grown. In this issue of Simulation in Healthcare, Waisel et al2 describe a research use of simulation for assessing how clinicians deal with “Do Not Resuscitate” (DNR) orders and preferences for patients who are facing and then undergoing surgical procedures. Although many review articles on the topic of DNR in the OR have been published over more than 15 years (several in the past few years)3–8 the implementation of the principles described therein is thought to be patchy at best. Simulation provides a crucial window on clinical processes and clinician performance. I have long described the complementarity of simulation with other methods of investigating performance. Recently, Mcintosh9 named this concept “Gaba's Jigsaw” alluding to the notion that each of the windows on performance is another piece of a larger jigsaw puzzle. The study of Waisel et al2 adds considerably to our knowledge of the issues and problems surrounding the way that clinicians handle the preferences of patients with DNR orders, because it allows the study of what people actually do (albeit in simulations) rather than what they think should be done, or what is reported in retrospect. Each method of assessment has strengths and weaknesses. Prospective nonbiased assessment of clinical process and clinician behavior in real case situations would clearly be the best. For some questions this can be accomplished by making recordings of data (through electronic medical records—the medical equivalent of aviation's “black box”) or even sight and sound (through embedded microphones and cameras—the equivalent of the “cockpit voice recorder”) of many cases to capture those of interest for detailed study. Yet, few patients coming for anesthesia and surgery have DNR orders so one would have to capture very many cases to get a fair sample of such situations (or else to have either an expert observers or a camera crew waiting patiently and immediately available for the few cases that come along). And—I can attest to this personally—the medicolegal, ethical, and logistical issues raised by filming in the clinical environment are daunting. Moreover, each such clinical situation is different, making it hard to see the spectrum of performance of clinicians responding to a “standardized” challenge. Case reports of real cases also offer a different window on performance. Such cases are actual clinical situations and only ones of salience would be brought to researchers' attention. However, the retrospective nature of these reports limits the information available, and memory recall of complicated cases is often faulty and subject to various hindsight biases. Moreover, there is likely to be a selection bias. Reporters are only likely to report cases in which there were significant problems and not those for which care processes were faulty but whose results were unremarkable. Even without these limitations, there is still the problem that all the cases are different making it is hard to generalize from the reports. Simulation therefore closes some of the holes left by the other techniques while possessing some of its own. Each case scenario is standardized so all clinicians face the same challenges and conundrums; these can be as difficult as desired. Observation can be prospective and recorded in a nonbiased fashion, typically using audio and video capture. Recordings allow for analysis by multiple observers or in multiple passes. Observers can be chosen with special expertise or training if desired. The big limitation is that simulations are not real cases, and all participants know that this is true. This hole can be closed—or another piece of the puzzle added if the simulations can be conducted such that the clinicians are unaware that it is not real. An example of such an approach when unannounced clinic visits by standardized patient actors are used for quality assurance or research purposes in the outpatient clinic setting.10–12 For acute care treatment situations, where actors cannot be used, such surreptitious simulations would require an “android” so good that it could be slipped into the clinical environment as if it was a real patient. This technology does not exist now and it may never be available. Thus, a fundamental limitation of acute care simulations for the foreseeable future is that clinicians know they are simulations. How is this likely to affect their performance? On the one hand, when they know that the “patient” is not a real patient, we can imagine that they will not take the situation seriously and will not perform all the actions as they would with a real patient. This can sometimes manifest itself as unusually sloppy or “cavalier” behavior, or clearly acting like it is all a game. Conversely, we sometimes see “hypervigilance” where participants later admit to doing more than they usually would do or to jumping too quickly on signs of possible trouble. Often this is because, knowing it is a simulation, they were expecting problems to occur and they acted on the first glimmer of abnormality. Other times they say that it is because they knew they were being filmed and thus felt that they should do everything “by the book” even if the book is usually honored only “in the breach” (as Shakespeare put it). One thing we learned in our studies of fatigued versus rested anesthesiologists13 is that it is very hard to reproduce the “motivational structure” of real work—the incentives and disincentives and culture of the workplace. This is a real problem for investigating performance in situations that depend on subtle judgments, negotiation between personnel, and the concerns for personal advancement, administrative discipline or risk of litigation. Waisel et al report some behaviors that seemed to stem from group interactions (usually, because of the script, pushing them toward more invasive and long-term interventions) but it is hard to know whether these suggestions would have occurred in a real case, or how the arguments would actually play out in a real operating room with senior surgeons and anesthesiologists who have their own interpersonal histories and intra and interdepartmental politics to consider. Again, without the undetected surreptitious android it is pretty much impossible to study such issues with high generalizability. If we assume that this simulation study of DNR situations does provide accurate data, what are we to make of the results? It should not be surprising that what people actually do in complex situations is not quite what we hope or expect that they would do or what is recommended in the various excellent review articles about the issue. Even for clinical care issues with widely accepted protocols, the implementation of the protocols is patchy. For ethical issues for which different clinicians likely have different personal opinions, high variability is even more expected. Nonetheless, certain features stand out as failures of our current system. Even knowing that they were being filmed, a substantial number of clinicians never even discussed the DNR issues with the patient to confirm what was written in the chart. Questionnaire data showed that 50% of participants were mostly or totally unfamiliar with ASA guidelines on the topic. This is disturbing especially because the participants in this study were faculty anesthesiologists at quarternary care hospitals who are responsible for teaching trainees how to practice, and who also have a more significant likelihood of encountering complex care for terminal patients than might be true in community hospitals. Waisel et al suggest that more education on these issues will help to increase the frequency and depth of discussions of the issues in the preoperative interviews; certainly for those who acted out of misapprehension of current protocols (eg, “automatic suspension of DNR orders”) or of total ignorance of the topic, education may at least sensitize them to the issues or correct their mistaken beliefs. Perhaps, training and practice in the conduct of difficult conversations, and more specifically about conducting preoperative interviews about DNR issues, would be more effective in ensuring thorough preoperative planning. It has become commonplace to have patients complete “satisfaction” surveys periodically after clinical encounters. Should we target some of these at the process of managing ethical issues including those of DNR in the perioperative period? Should we set benchmarks for performance to provide incentives to follow established guidelines? Given the nature of the reasoning used in the response to the dynamically evolving situation in these simulations, it is clear that establishing a goal-directed plan for patient care is only a first step, and that many pitfalls exist in actually implementing the plan. The results of a study by Waisel et al highlight the need for more attention to these issues, and also suggests the possible benefits of using simulations with both standardized patient actors (for preoperative interviews) and multiclinician scenarios of evolving catastrophes to address the optimum ethical management of patients with clear-cut goals and preferences for limiting resuscitative efforts. Thus, I believe that this study provides an important demonstration of how simulation can be a powerful tool for delineating strengths and weaknesses of our systems and processes including the handling of delicate issues of professionalism and ethics. It is good for us to know how clinicians really behave (or at least how they are likely to behave) rather than to merely assume that our typical “good clinicians” already know everything about relevant protocols or principles of ethical care and can readily implement them when things turn sour. ACKNOWLEDGMENTS The author thank Associate Editor Barry Issenberg for raising some good points that are now embedded in this editorial.