Team Performance Assessment in Healthcare

It has been widely recognized in the patient safety literature that team performance is crucial to providing safe patient care, and “team failure” has become a key factor addressed by many system-based interventions to improve patient safety.1–3 Teamwork has become a focus of medical education standards,4,5 and the simulation community has responded to this by designing and implementing simulation-based team training programs for many specialties and increasingly for multispecialty teams.6–9 However, if team performance cannot be assessed accurately, efforts to define specific training needs and to improve team performance may be futile. Thus, medical educators and human factors researchers are now facing the challenges of team performance assessment—some of which are known from other safety critical industries and some of which may be unique to healthcare. This edition of Simulation in Healthcare features an article by Rosen and colleagues outlining best practices for team performance assessment.10 The authors are to be congratulated for providing a concise overview of the challenges of team performance assessment in the context of simulation-based training. Building on an extensive body of research on team performance assessment in various safety critical domains, they provide a list of 11 “best practices,” recommending that these best practices should also guide simulation-based team performance assessment in healthcare. The future success of simulation-based research and training in healthcare most likely will be determined by how successfully the simulation community responds to the challenges associated with performance assessment of both clinical and nontechnical skills (ie, the cognitive and social skills required in any operational task involving decision-making and team work11) on an individual and team level. I would like to explore the issue of how the current (published) practice of team performance measurement in healthcare does compare with the best practices described by Rosen and colleagues (Table 1).Table 1: Best Practices of Simulation-Based Team Performance Assessment (Rosen et al., 2007)10The authors begin their list of best practices by stating that team performance measurement needs to be grounded in theory because the theoretical model referred to defines the elements of a measure (eg, backup behavior and leadership can be elements of a team performance measure) (see best practice #1). However, healthcare researchers and educators may find it difficult to choose because theoretical models focusing on different aspects of team performance are still under development.12 Most conceptual models of team performance are presented within the framework of inputs (eg, characteristics of the individual, the task, and the organizational structure), processes (eg, communication, decision making), and outputs (eg, quality and quantity of products or services, improved knowledge of team members) (IPO models).12,13 The main advantage of an IPO model is that it allows for temporal considerations, describing a team process by which inputs are transformed into outputs that then serve as inputs into another process. Thus, the focus of performance assessment is shifted from the output to the process by which an output is achieved. Of course, to provide a complete picture of team performance, both aspects need to be included (see best practice #8). Empirical data from healthcare looking at the relationships among inputs, processes, and outcomes14–17 will contribute to an increasingly complete picture of team performance in healthcare and beyond. To be relevant in practice, measures defined on the basis of a theoretical model need to capture critical competencies and thus have to be adapted to the types of teams and team tasks studied (see best practice #3). The research on nontechnical skills in anesthesia,18 surgery,19 and neonatal resuscitation,20 where many aspects discussed by Rosen and colleagues were considered carefully, is an excellent example of scientific developments that can be used for research and training purposes during simulation as well as in clinical settings. Evaluation results for anesthesia show, for example, that clinicians believe that the behaviors included in the rating system are observable (see best practice #6) and that such systems do facilitate debriefs of trainees11 (see best practice #11). In the context of simulation-based training, best practice #2 “Design measures to meet specific learning outcomes” and best practice #5” Link measures to scenario events” are particularly important. Some simulation experts in healthcare do consider these issues when designing simulation scenarios, but many do not systematically include performance assessment issues from the start. Still, performance assessment will take place, and feedback will be provided after each scenario. This is often based on a set of rather generic crisis resource management skills that have proven very useful for training purposes in many safety critical domains. For research purposes, however, the fact that clinical as well as team performance may vary over time21 presents a critical challenge. Defining specific team performance measures for each critical event in a scenario may provide a feasible solution to this methodological problem. But, which team process variables can be used to distinguish and predict effective versus ineffective team performance for a specific critical event? In general, outcome standards for technical performance form the background against which team processes criteria are established. In many healthcare settings, there is still a lack of agreement on clinical outcome standards and their measurement. Thus, many studies on team performance in healthcare focus on the similar scenarios.14,16,17,21,22 Continued progress on clinical performance measurement will be critical for further progress in team performance assessment in healthcare. The basic psychometric challenges of assessing human performance in a complex activity are uniform across domains and concern technical as well as nontechnical aspects of performance (see best practice #9 and #10). Byrne and Greaves reviewed instruments for performance assessment in anesthesia and concluded that only few studies have specifically addressed validity and reliability issues.23 Focusing on team performance measurement, a recent study found that many characteristics used to measure team performance during obstetric emergencies could not be scored with sufficient reliability although expert raters did agree on overall team performance scores.24 Addressing the methodological challenges of simulation-based performance assessment as well as the practical aspects of implementing team performance measures in simulation-based training remains a critical challenge for the future. From a human factors perspective, major issues in team performance assessment that remain to be resolved include, for example, the relative importance of individual and team skills in team performance (see best practice #4). Refined conceptual models and empirical results will help to determine the appropriate unit of analysis (individual versus crew/team) for a given research question and establish methodological standards for integrating measures focusing on the individual and on the team level where necessary. Focusing on team training efforts, the question remains whether there are generic or transferable team skills that will improve team performance when taught to individuals or whether highly performing teams possess team skills over and above the individual skills. These issues are particularly important for teams in healthcare settings where individuals with diverse professional backgrounds come together to form a team for a shift or even a single case. From a systems perspective, it also has to be concluded that simulation-based training alone will most likely be insufficient to change the way teams work together in clinical practice. We need to know more about characteristics of team task design, safety-related attitudes, and factors in the work environment that may hinder optimum team performance (see best practice #4). I believe that Rosen and colleagues have provided an important contribution to the simulation community's discussion on best practices for team performance assessment that will hopefully stimulate further scientific endeavor in the field of simulation-based performance assessment in healthcare. Performance assessment is at the core of educational and human factors research in all domains of medicine and key to any systematic training program. Thus, performance assessment has been discussed since the beginning of simulation in medicine, and in recent years, important work has been published on the assessment of team performance or nontechnical skills in healthcare. However, so far, there is no common ground on how to measure team performance in healthcare, and theoretical as well as methodological challenges remain. To address these issues in a way that takes into account the specifics of healthcare while drawing on experience from other industries will probably become even more important in the future, because ultimately, research results that identify the characteristics of effective team performance and the most effective training strategies are critical to improve patient safety.