What counts as evidence in an evidence‐based world?

Contemporary health care has become preoccupied with evidence. “Evidence-based practice” has permeated several (if not all) health care professions and most aspects of service provision. That fact is evident in the many articles published in this issue of the Journal of Evaluation in Clinical Practice. It is good that decisions about diagnostic tests, management of care, and the organization and allocation of health care resources should be based on evidence. I doubt anyone would believe (or admit) that it should be otherwise. The mere suggestion that health care should be “evidence-based” implies that health care activities can also be “not evidence-based.” Thus, it begs the question as to what makes something “evidence-based” (and for that matter, what makes something “evidence”?). What are the alternatives to “evidence-based”? The answer to these questions has significant implications on which interventions are selected for practice and how they are studied in an evidence-based world. As health care decisions are unlikely to be arbitrary, it is not unreasonable to suggest they are usually based on some kind of evidence. When controversy arises as to the value of a health care intervention, it is often about disagreements regarding the quality of that evidence. When we speak of evidence, it is relative to a specified hypothesis; some observation is not by itself evidence. Munro et al1 suggest that in general usage, evidence “means information that is used to provide support for a conclusion” (p. 1). That is helpful, but perhaps we can expand on this suggestion. Our basic intuition about evidence is that it makes a difference to what one is warranted in believing: If E is evidence for the hypothesis H, then E makes it more likely that H is true. E provides empirical support for H, although coming to a position on the degree of support would require a filter to allow us to set a value on the different pieces of evidence that make up the total evidence in support of hypothesis H and resolve any tension between these pieces of evidence. With that in place, a working theory of evidence at a minimum should satisfy two conditions: (1) It should stipulate what kinds of facts/observations are needed to evaluate a hypothesis (relevance condition); and (2) it should give us guidelines or criteria that tell us how to evaluate the hypothesis given the facts that we have collected (assessment condition). * It is these criteria that inform us how much support of what kind we need in order to achieve a different degree (i.e., relative to not having the fact/observation) of warrant for that hypothesis. Evidence Based Medicine (EBM), coupled with the GRADE framework,3, 4 can be considered a theory of evidence that satisfies these two conditions, although some may argue that is does so in an unsatisfactory way. Furthermore, a good theory of evidence should be sensitive to the fact that warrant for a hypothesis is not an all or nothing affair: A hypothesis can be strongly warranted by the evidence or weakly warranted by the evidence. We can have good reasons to infer a hypothesis or not good reasons. EBM is sensitive to that—its evidence hierarchy gives practitioners a tool to assess the degree to which the evidence warrants a hypothesis. It is likely for that reason why EBM has gained so much interest among health care stakeholders. Sackett et al5 famously described EBM as “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” (p. 71). That quote begs the question as to what is meant by the “best” evidence. In an EBM framework, “best” is determined by the aforementioned evidence hierarchy. The hierarchy was established on methodological grounds with respect to perceived potential bias in the estimate of effect. Advocates of EBM use terms such as “quality,” “confidence,” or “certainty” when referring to evidence and in some cases (e.g., GRADE framework) provide a rule set for determining what one ought to believe regarding the estimate of effect of an intervention.6 Thus, EBM is a normative stance on evidence, at least with respect to clinical interventions. In the strictest sense of EBM (or perhaps a fundamentalist approach), one does not have evidence that an intervention has a therapeutic effect unless one has high-quality randomized controlled trials (RCTs) demonstrating such (in some cases, observational methods that yield large effect sizes and control for important confounding variables are an acceptable substitute). † Supporters of the EBM movement might object to such a characterization. However, it does point to a potential demarcation problem—attention to methodological rigour is important, but it is not clear what rigour is required to merit evidence for clinical practice (or when is a study so poor that it cannot be considered evidence). It has been suggested to one of the authors on several occasions that EBM is not strictly the application of a pre-specified hierarchy when reviewing the evidence for or against an intervention (and then basing practice off that assessment). However, those who criticize evidence hierarchies are given the illustrious distinction of being “anti-EBM,” implying that evidence hierarchies are at least a necessary component of EBM practice. If it is the case that EBM is more than an application of an explicitly defined hierarchy, it is not clear what else distinguishes its practice as uniquely “evidence-based” from other approaches to clinical medicine. The use of inoculation for small pox and penicillin both predated the EBM movement and the clinical trial, and they most certainly were “evidence-based” practices. Likewise, few would object that the observations of Jenner and Fleming count as evidence supporting the hypotheses that inoculation prevents small pox and that penicillin treats bacterial infection, respectively. ‡ Clinical decisions based on similar science today would certainly be evidence-based, irrespective of one's views about hierarchies or the value of a clinical trial relative to information that is deemed to be relevant to the case at hand but was procured in another way. It is worth mentioning that the defined evidence hierarchies utilized in clinical medicine are peculiar when compared to other sciences. The physical sciences do not use such hierarchies (or at least they are not explicitly defined), and yet no one is suggesting decisions using principles derived from those sciences are not evidence-based. We will not dispute that for any given question, some kinds of observations will have more evidentiary value than others—that is true for all sciences. A staunch supporter of EBM may then argue that what distinguishes EBM practice from other approaches is that it advocates for the application of the “best” evidence (whatever it may be) when making decisions about clinical practice. However, as only the strictest of relativists would argue in favour that all observations should be weighted equally regarding its value as evidence for or against a hypothesis, there is nothing special about the EBM view that attention should be given to the best evidence beyond a normative claim of what counts as the best evidence. The EBM movement was important in that it raised awareness for the need in many contexts for better evidence to ground clinical practice. Unfortunately, scholarship supporting EBM has not advanced our understanding of evidence as a concept. As our understanding of evidence is incomplete, we require theory that allows for further examination of the concept. The normative framework advanced by EBM, on the other hand, hamstrings such examination by encouraging practitioners to dismiss anything that falls outside the framework (including devaluing the utility as evidence of anything that is not derived from RCT methods) and marginalizing differing views as being “anti-EBM” (i.e., “anti-evidence-based”). Trial law provides an interesting perspective on evidence that may help advance our conceptual understanding of evidence in the context of health care practice. Court proceedings and the determination of fault are certainly evidence-based practices. Again, an observation is not by itself evidence, nor are observations purported to support a hypothesis raised in a legal case automatically accepted as evidence. Although there are standards and precedent for evidence in a trial (i.e., what is considered relevant), the admissibility of some observation as evidence for or against a hypothesis is decided through deliberation (i.e., a form of assessment). Judgment in that deliberation is openly acknowledged—courts are presided by a literal judge who makes the final determination about relevance and whether such information will be prejudicial to the defendant's right to a fair trial. § In a similar vein, Munro et al1 have advocated for more attention to the role of deliberation when participating in evidence-based policy and moving away from a technocratic approach (where tasks are proceduralized and the role of individual expertise is reduced). A deliberative process is a rejection of the dominant narrative “where references to ‘what works’ are made as if the findings of an RCT can be readily generalized” [1; p. 146]. Evidence-based practice is an admirable goal—who would object to basing important decisions on evidence? What is problematic in our quest to align health care practices with evidence is that we do not have a clear conceptual understanding of what makes an observation relevant as evidence, and how to judge the extent to which relevant observations provide support for or against a specified hypothesis (i.e., assessment). The EBM movement, despite its importance in highlighting that we should be vigilant about what serves as the basis for belief about health care practices and therapies, has sidestepped deep philosophical problems of evidence and has instead adopted a normative stance. That is not to say that such a normative stance has been unhelpful in improving care for patients. However, a normative stance can lead to commitments to specific kinds of observations as evidence to support a hypothesis that may preclude seeking or attention to observations that give us a more correct impression of whether that hypothesis is warranted. For example, privileging the evidence value of trial observations over mechanisms and/or clinical experience may be helpful for hypotheses related to populations (assuming there is high fidelity between the trial and target populations with respect to important contextual features), but can be problematic for a hypothesis related to an individual patient in the clinical encounter.10 Unfortunately, clinical scientists, health care managers, and service providers often use such information as evidence for the wrong question (observations about what works on average is used as evidence of what will work for you). Oftentimes that is due to either not appreciating or not bothering to acquire the kinds of observations that can be sought at the individual level, because such observations are considered to be of low evidentiary value on the evidence hierarchy. It is such concerns with normative frameworks that have led many contemporary philosophers of science to adopt an evidence pluralism stance.11 Evidence-based practice relies on our ability to identify when an observation supports a hypothesis and when such support allows us to achieve a sufficient degree of warrant for that hypothesis. More attention to a theory of evidence will better equip us for that task. The normative framework for evidence promoted by the EBM movement has become the dominant view among health care stakeholders. That dominant culture can promote in some (or perhaps many) a proclivity towards certain kinds of experimental evidence (in particular, RCTs) for fear of being seen as not scientific or not caring about evidence. A commitment to normative stance on evidence (or fear to not adopt the dominant one) hampers an advancement in our conceptual understanding of evidence. Without such understanding, evidence is reduced to nothing more than a term that we use to justify our decisions. The health care community has become obsessed with evidence, and yet we often give little regard for what it means.