Summative Assessments Using Simulation Requires Safeguards

This is the authors' abstract. We don't add key points for this paper.

To the Editor We read the article by Rochlen et al1 with great interest. They conducted a survey of anesthesia residencies across the country to elicit attitudes on simulation-based education. Their survey received a robust response rate from program directors. Their results found that the overwhelming majority (89%) favored the strengths of simulation as an educational tool for resident training, and they agreed that standardization of components of a simulation curriculum would be advantageous. Simulation covers a spectrum of modalities and applications. Rochlen et al1 did not specify in detail the simulation activities, and thus, it is difficult to know which aspects of simulation were used for assessment. Even so, we foresee the growing use of simulation as a tool for summative assessment. On the basis of our experiences and current literature, we express caution to educators to ensure that certain safeguards are in place before incorporating the summative assessments into their simulation curriculum: Educators need to fully disclose and distinguish whether learner performance will be assessed or tested during the simulation exercise. Students already approach the simulation environment with apprehension, knowing that their skills will be challenged and their performance will be observed by both faculty and peers. One of the greatest values of simulation lies in the psychological safety net that lies beneath the clinical challenge. Students have the opportunity to be innovative and creative in finding solutions, with the recognized support of their peers and instructors, without the threat of adverse patient outcomes. Proper disclosure of the testing environment is critically important to prevent systematically poisoning the atmosphere for the learning sessions and inadvertently creating learner aversion to simulation-based education. Simulation should not be a single point of failure for any high-stakes summative assessment. Simulation may be a useful component of both formative and summative assessments to be used in combination with other modalities and approaches to overall student assessment. Students should have the opportunity to engage in multiple simulation-based encounters because variability in learning performance can be high across simulation activities. They should also be able to retest their performance if it does not meet expectations.2 Evaluation metrics chosen by educators should at the least be sound and ideally be validated. We believe simulation-based assessment provides a unique window that no other current measure provides.3 Various tools exist, including Anaesthetists’ Non-Technical Skills (ANTS).4 Still, caution must be exercised because even one of the most used behavioral marker systems, the ANTS system, “is not recommended for formal summative assessment.”4 Improper and careless implementation of simulation-based summative assessment in anesthesiology residency will likely endanger the safe-teaching environment that we, simulation instructors, have worked hard to preserve. The next logical step in moving our field forward is to expand the study and survey by Rochlen et al1 on how formative and summative assessments are currently being implemented in anesthesia training programs. We can then determine strategies for creating the best way to integrate simulation in assessing learner performance. Louise Y. Wen, MDDepartment of Anesthesiology, Perioperative and Pain Medicine,Stanford University School of Medicine, Stanford, CA[email protected] David M. Gaba, MDDepartment of Anesthesiology, Perioperative and Pain Medicine,Stanford University School of Medicine, Stanford, CAVeterans Affairs Palo Alto Health Care System, Palo Alto, CA Ankeet D. Udani, MDDepartment of Anesthesiology, Duke University School ofMedicine, Durham, NC