Healthcare education research: The case for rethinking hierarchies of evidence

With global demand for healthcare services increasing exponentially, it has never been more important to demonstrate clinical effectiveness to achieve the best outcomes for patients, clients, service users and carers, while also ensuring value for money. The role of research is to inform these outcomes and identify the efficacy of specific interventions. A range of hierarchies of evidence have been developed to characterize the strength of this evidence and, therefore, by implication, its usefulness for evidence-based/evidence-informed policy, decision-making and practice. Healthcare educators are not immune from similar considerations regarding the effectiveness and outcomes of the education they design and deliver. All parties involved in healthcare education are increasingly interested in the evidence of its effectiveness, that it enhances patient/service user outcomes and care and represents a good return on investment. Educators must, therefore, be cognisant of the needs and expectations of a diverse range of stakeholders, including healthcare service users, employers, education commissioners and students. Randomized controlled trials (RCTs) have traditionally been regarded in certain disciplines, especially medicine, as the gold standard research methodology for generating robust evidence of effectiveness. However, conducting RCTs in educational settings is complex and fraught with difficulties with many variables to capture and control. In addition, we must acknowledge the needs of our students on their journey to becoming competent and confident practitioners and therefore other, more generic educational outcomes, such as the student experience and student satisfaction ratings, are also important. RCTs therefore may not measure either the specific or the more general educational outcomes that are important to healthcare educators. In this paper, we argue that the dominant view that only RCTs can generate “gold” standard evidence is flawed in the context of healthcare education research. As context and process are both crucial in any education research, undertaking RCTs (or other forms of experimental research) is problematic. We argue that the definition of “gold” standard evidence should be determined by the kinds of research questions asked and the appropriateness of the research design, rather than being dictated by a one-size-RCT-fits-all approach. This, in turn, raises questions about whether conventional hierarchies of evidence serve a useful purpose for healthcare education research. We acknowledge there may be no easy answers but hope to trigger an important and timely debate. Traditional hierarchies of evidence assume that some forms of evidence are stronger than others. It is a widely held view in healthcare that the “best”, most robust evidence is derived from well-executed, up-to-date systematic reviews and meta-analyses of multiple, well-designed, large-scale RCTs and from RCT themselves. The Cochrane Library and the Cochrane Database of Systematic Reviews illustrate the contribution they can make to healthcare policy, decision-making and clinical practice. However, the methodological challenges associated with RCTs have been acknowledged over the decades (see, for example, Greenhalgh et al., 2003) and continue to be discussed. For instance, in 2018 the methodological debates and challenges associated with RCTs are to be the focus of a Special Issue of the journal, Education Research. Experimental research is particularly well suited to laboratory research where the environment can be tightly controlled, so the researcher can be reasonably confident (within the confines of statistical significance) that factor X is responsible for outcome Y. With tight control and elimination of bias, it is assumed that confounding variables are eradicated as possible causes of recorded differences in outcome. However, conducting experimental research in the complex, real world of education practice is complex and challenging as highlighted by the Medical Research Council (MRC) guidance on developing and evaluating complex interventions (MRC, 2006). The MRC defines complex interventions as interventions with several interacting components, which present several challenges for researchers. These include the difficulty of standardizing the design/delivery of an intervention, sensitivity to the context, the organizational and logistical difficulty of applying experimental methods, the length and complexity of the causal chains linking intervention with outcome and ethical issues. RCTs are also expensive and time-consuming to conduct, so the evidence generated may not necessarily keep pace with innovations in practice and the rapidly changing healthcare landscape. What does all this mean for healthcare educators aiming to provide evidence of education effectiveness? Greenhalgh et al. (2003), p. 145) argued that: “the linear and formulaic link between evidence and practice implicit in evidence-based medicine [has] proved inadequate for the complexities of educational research”. Moreover, this gold standard of evidence could only ever be generated using an RCT when a specific intervention is evaluated in a tightly controlled environment which, paradoxically, would limit the generalizability of the findings. This then raises the question about whether it is ever possible to generate the highest levels of evidence (as traditionally defined) when we simply do not know what the: “active ingredients” of an educational intervention are. Therefore, a search of the Cochrane Database revealed that education interventions are notably missing. Presumably, this is because few, large-scale RCTs have been published and the so-called “gold standard” evidence does not exist, making it challenging to undertake any form of systematic review. There is, however, one notable exception—the Cochrane review of the effectiveness of interprofessional education (IPE) compared with profession-specific education. The first systematic review, published by the Cochrane Collaboration in 1999, indicated there was no high-level evidence available that met the inclusion criteria. When this review was updated in 2010, there were six studies, involving RCTs or controlled before-and-after studies that met the inclusion criteria. By 2013, this figure had increased to 15. However, despite the increased number of studies in this most recent systematic review, the reviewers concluded that it was not possible to draw any generalizable inferences about the benefits of IPE on patient outcomes or healthcare process outcomes due to the heterogeneity of the educational interventions and the context-specific nature of the studies. Indeed, one of the recommendations was the need for research studies with “qualitative strands” examining the processes of IPE. The long and complex chain of events from teaching in a university to the reality of healthcare practice, raises the question of whether it will ever be possible to demonstrate a direct link between the two. Given the importance of context in education research and the challenges outlined above, we argue that it does not make sense to strive to generate gold standard evidence as judged by conventional evidence hierarchies. The influence of the “real world” context cannot be eliminated and so RCTs are unlikely to yield data that are meaningful or relevant beyond the study context, with an inevitable knock-on effect on attempts to review systematically any accumulating evidence. However, we cannot simply and conveniently put aside the notion of levels or strength of evidence without proposing an alternative. In a paper entitled “On hierarchies, malarkeys and anarchies of evidence”, Booth (2010) suggests the adoption of typologies of evidence. Rather than focusing on internal validity (as is the case in evidence hierarchies), typologies of evidence align types of research question with different types of research methodology—the prime concern being methodological appropriateness and research quality rather than the supremacy of particular designs. Several evidence typologies have been published. Upshur, VanDenKerkhof, and Goel (2001), for example, proposed a model of evidence that combines measurement and meaning. Measurement, in the context of positivist research, values precision, accuracy and generalizability. Meaning, on the other hand, focuses on contextual dimensions: “the salient features of this form of evidence are its narrative structure and concern with the interpretation of meaning rather than quantities or properties of objects. Epistemologically, it is also rooted in empiricism, but not necessarily quantified reasoning” (p. 93). If both measurement and meaning are afforded equal weighting, then a broad range of research approaches are necessary to generate the evidence base to underpin policy, decision-making and practice. This is an argument for a mixed-methods approach. Petticrew and Roberts (2003) identified eight key features in their typology of evidence: effectiveness; service delivery; salience; safety; acceptability; cost-effectiveness; appropriateness; and satisfaction. They matched these against different research approaches, creating a framework that highlights the crucial importance of selecting the most appropriate methodology to answer the research question (Table 1). Effectiveness Does this work? Does doing this work better than doing that? Process of service delivery How does it work? Salience Does it matter? Safety Will it do more good than harm? Acceptability Will users be willing to or want to take up the service offered? Cost-effectiveness Is it worth buying the service? Appropriateness Is this the right service for these users? Satisfaction with the service Are users, providers and other stakeholders satisfied with the service? Glasby, Walshe, and Harvey (2007) propose a typology of evidence based on three types of evidence that practitioners might seek, use and act on when considering “what works” in health and social care: theoretical evidence, empirical evidence and experiential evidence (Table 2). All three kinds of evidence are important, so no hierarchy is implied. Similarly, Larner (2004) proposed a more diverse evidence base, arguing for a “both/and” stance in the field of family therapy. This involves the use of RCTs where appropriate and also other types of evidence. He calls this “paramodern” where “narrative and science, qualitative and quantitative research, modern and postmodern perspectives sit together as a necessary tension, sharing an investigative, ethical and pragmatic curiosity about what is helpful in the difficult world of therapy” (p. 34). This approach is surely equally applicable to fields such as healthcare education. While healthcare educators may aspire to generating gold standard evidence, as defined by traditional evidence hierarchies, we need to acknowledge that although they may have a role to play, RCTs/experimental research alone is not the most appropriate means of evaluating complex education interventions. Conventional evidence hierarchies are not, therefore, the most appropriate framework to judge the quality of research evidence generated by healthcare educators and, in this paper, we propose evidence typologies as a more suitable alternative. Given the dearth of evidence of the effectiveness of healthcare education interventions and the current outcomes-based/value for money culture in public services, a shift in mind-set and perspective is urgently required. This means stepping away from traditional evidence hierarchies and an emphasis on RCTs and outcomes to focus on research methodologies, such as realist evaluation, case study and participatory, constructivist research approaches, that are designed to achieve a more nuanced understanding of the education process and context. It might also mean examining education research in different disciplinary sectors including school and further/higher education. Educators, researchers, commissioners and policy makers all need to embrace this shift in emphasis. It is important to acknowledge that this debate takes place against a political backdrop of international university rankings based on a range of metrics, including research and teaching excellence. Although there is ongoing discussion about whether rankings encapsulate the breadth and depth of what universities can and should offer, they do drive institutional behaviours. Perceptions of research quality are part of this wider focus on league tables based on university rankings, with conventional evidence hierarchies still influencing the way where evidence is viewed and reviewed in some quarters, how judgements are made about research quality and funding allocation. This political context may stifle the development of more innovative approaches to healthcare education research. So, going forward we need a shift in conversations in healthcare practice settings, faculties and universities that value equally a range of methodological approaches to generating evidence, which are of high quality and have the potential to enhance healthcare education. This, in turn, requires different forms of literature review methodologies to bring together and combine the range of evidence in a robust and transparent way as a basis for decisions about education policy, practice and funding. As Maxwell (2012, pp. 145–146) contends, evidence “can't be assessed in context-independent ways, but only in relation to the particular question and purpose to which it is applied and the specific context in which these are investigated. …Any attempt to establish a context-free hierarchy of kinds of evidence based entirely on the methods used to create that evidence, as proponents of “evidence-based” approaches typically do, is inevitably flawed”. This paper developed out of a workshop at the 27th International Networking for Healthcare Education Conference in Cambridge, UK, in September 2016. The authors wish to acknowledge the contributions of the workshop participants—Ms Ann Price, Canterbury Christ Church University, England; Ms Christine Sommers, Universitas Pelita Harapan, Indonesia; Dr Kirsi Talmin, Helsinki Metropolia University of Applied Sciences, Finland; Dr Mark Wareing, University of Bedfordshire, England; Dr Tessa Watts, Swansea University, Wales—who have commented on various drafts of the paper. We would also like to acknowledge our “critical friend”, Gary Rolfe, Emeritus Professor of Nursing, Swansea University, Wales, for his characteristically valuable, critical commentary on the penultimate draft.