Best-Evidence Synthesis: Why Less Is More

In his response to my proposal for an alternative to meta-analytic and traditional reviews (Slavin, 1986), Bruce Joyce has made the exact misinterpretation of what I am proposing that I most feared readers would make. Therefore I am delighted to have an opportunity to clarify what I meant by and what I did not mean by it. One thing I did not mean to suggest was that reviewers establish a priori inclusion criteria before searching the literature. Rather, my position is very close to that which Joyce proposes: Find everything, then organize it carefully. The first step in conducting a best-evidence synthesis is to conduct a broad-scale preliminary literature search in order to fully understand the meaning and quality of the evidence. Only then am I suggesting formulating inclusion criteria. Also, I am proposing that the evidence excluded be described in some detail, so that the reader, to use Joyce's term, can enter the thought processes of the investigator and understand the consequences of whatever inclusion criteria were applied. Actually, I would have no objection if a reviewer obtained every study on a particular subject and then partitioned the evidence according to well-defended criteria. What mostly differentiates bestevidence synthesis from metaanalysis is not issues relating to inclusion, but rather the requirement is a best-evidence synthesis that the studies on which the argument depends be identified and described in enough detail to allow the reader to understand what the body of research really investigated and what it found. Very few meta-analyses discuss more than one or two of the dozens or hundreds of studies whose results they synthesize. The issue of a priori inclusion comes up only because it is a practical impossibility to discuss everything in a l rge literature, so we must have some means of deciding what is worth discussing for a given purpose. Every review makes one or more principal conclusions. What is he nature and quality of the evidence on which these conclusions rest? The reader has a right to know.