The Importance of Data-Selection Criteria: Meta-Analyses of Stream Predation Experiments

The value of meta-analysis in ecology hinges on the reproducibility of patterns generated by quantitative synthesis. Meta-analysts will vary in the criteria they use to screen studies and select data within studies, even when addressing exactly the same question. We summarize some of the many decisions that an ecologist must make in deciding what data to include in a synthesis. We then show, using multiple meta-analyses taken from the same literature on stream predation experiments, that meta-analytic conclusions can be colored by selection criteria that are not specifically a function of the relevance of the data. As a consequence, we recommend that meta-analysts perform several meta-analyses using different selection criteria to examine the robustness of reported findings. We also advise ecological meta-analysts to minimize use of selection criteria that are based on judgments of study quality when extracting data from the literature, because of the potential for unconscious bias. The influence of quality criteria on patterns in the data set can then be examined empirically. Our comparisons of mean effect size, for studies included vs. excluded on the basis of “quality” criteria, provided no evidence that rejected studies were aberrant or more variable than “acceptable” studies. One result of excluding such studies was a loss of statistical power. We urge ecologists to be more explicit about how data are selected for a meta-analysis, to examine the robustness of the patterns they report, and to conduct meta-analyses to describe as well as to infer.