Emergency Ultrasound: Leveling the Training and Assessment Landscape

The use of ultrasound (US) by emergency physicians represents one of the major advances in clinical emergency medicine (EM) in the past two decades. In this issue of Academic Emergency Medicine, Amini et al.1 present the results of a survey of EM residency and US directors to ascertain the current methods of education, quality assurance, and competency assessment in existing US programs. They received surveys from 77% of residency programs, 87% of which have required rotations in emergency US. The survey findings reveal a broad range of assessment strategies and methods, with only about a third of programs reporting use of practical examinations and another third incorporating direct observation or standardized clinical encounters to assess US skills. The relative dearth of rigorous assessment programs and diversity of assessment practices is striking and highlight an educational problem that affects teachers, learners, patients, and care systems: how should competency be assessed in emergency US? This question is not new to US educators. The 1994 Model Curriculum for Physician Training in Emergency Ultrasonography deemed providers “trained” once they had completed 150 examinations and 40 didactic hours of education.2 Similarly, the American College of Emergency Physicians (ACEP) guidelines of 2001 and 2009, while acknowledging that learners achieve competency at varying rates, placed the emphasis on numbers of cases completed to demonstrate competency; recommendations range from 10 exams for procedures, to 25–50 exams for specific applications, to 150 total exams to earn “general” competency.3, 4 In 2007, competency assessment of residents' US skills became a focus of the Accreditation Council for Graduate Medical Education (ACGME) and the Residency Review Committee (RRC) for EM, as the EM Program Requirements were revised that year to include a mandate for documentation of assessment of resident competency in US, as well as two other procedures that were left to program discretion.5 In 2008, the Council of EM Residency Directors and leaders from the emergency US community weighed in, including in their model US curriculum for EM residency training programs a recommendation that US assessment should occur at least twice in a resident's training and should incorporate such methods as standardized knowledge tests, objective structured clinical examinations, the standardized direct observation tool, simulation, and video review to assess both US technique and image interpretation skills.6 The importance of US competency assessment was further emphasized by its inclusion as one of the 23 subcompetencies in the EM Milestones document (Figure 1).7 While recent publications have presented results of successful implementation of US assessment programs, suggested pathways and tools for assessment, and explored the use of simulation in US competency assessment, there is a paucity of literature to guide educators faced with an immediate need for an evidence-based assessment program.8-10 Educators turning to the medical education literature for assessment theory and strategies that may be translatable to US will initially be optimistic, as this topic has been extensively explored and there are myriad assessment tools described. However, that optimism fades somewhat when diving deeper, as one realizes that assessment is still a topic filled with debate, and despite the plethora of publications on the topic, many medical educators still feel ill-equipped to effectively assess clinical competence. At the heart of this debate are the proponents and critics of the competency-based education movement, with provocative arguments for both breaking down performance into observable competencies and considering performance as a global phenomenon or sum that may be incompletely measured even by accurately assessing its parts.11, 12 In a 2011 synthesis of the assessment literature, medical education leaders recommended that three critical issues be addressed to advance our assessment practices.13 The first issue cautions the educator community regarding the downsides of assuming that “‘the right’ list of separate competencies will be a sufficient operational definition of what it means to be a physician.”13 Inherent in this warning is a realization of the effect that assessment practices have on both learning and curricula. As US educators continue to develop competencies to assess, it would behoove us to remember that the skills that are easiest to measure are not always the most important, and “we tend to improve that which we measure.”14 The second recommends that educators move beyond “faculty development strategies that teach supervisors how to use [assessment tools] and address the administrative, professional, and cognitive barriers that impede supervisors' ability to formally codify and document their expert assessment of their trainees.”13 This does not imply that faculty development and assessment instruments are not needed, but rather that these interventions alone cannot overcome barriers to implementing assessment plans if these barriers are not explored and explicitly addressed. The final issue relates to feedback: “In addressing the mandate of our assessment programs to offer meaningful feedback to trainees, the community would do well to find mechanisms to engage residents in the assessment process such that they are able to appreciate and incorporate corrective feedback into their professional development.”13 The importance of feedback in improving performance is well recognized, but emerging theory supports that feedback effectiveness may be influenced by who initiates it, how the learner receives the feedback and his or her relationship with the feedback deliverer, and the nature of the feedback culture in the educational environment.15-19 Assessment should not only assure competency, but also be used to prompt feedback that aims to improve performance. Assessment without feedback presents a missed learning opportunity. Both real-time feedback given at the bedside, and delayed feedback given after images are reviewed in a comprehensive quality assurance program, are crucial to reinforce positive behaviors and allow the provider to learn from his or her mistakes. Given that US plays such an essential role in today's practice of EM, it is important that its application is standardized across residency programs.20 US should reproducibly demonstrate pathology regardless of training site location, just as the interpretation of ST-segment elevation on an electrocardiogram, right middle lobe consolidation on chest radiograph, buckle fracture on forearm radiograph, and dendritic lesions on fluorescein staining, are not subject to regional variations. Although the literature has begun to lay the groundwork for a standardized US curriculum and the development of US assessment tools, Amini et al.1 still describe a very heterogeneous training landscape. An ideal US competency assessment program would be fair, demonstrate validity evidence to support the interpretation of its results, be able to identify gaps in trainees' US knowledge and technique, be feasible and acceptable to learners and educators and would allow individual programs to innovate curricula and assessment tools to suit local needs while still ensuring uniformity of practice of the core applications of emergency US. Moreover, assessment should ensure that trained emergency physicians use US in a safe, accurate, and competent manner. Given the key role emergency US now plays in both clinical EM and graduate medical education, it is time to develop the educational and competency assessment literature of emergency US.