The Difference That Quality Makes

This paper addresses a traditional problem in the cumulation of scientific knowledge: the need to aggregate evidence from previous studies. Typically, the existence of many individual studies on a given topic makes such a step highly desirable, yet the procedures for "reviewing the literature" have not been developed in a methodologically rigorous manner. In particular, reviewers make many subjective judgments to distinguish those studies that are to be reviewed from those to be discarded because of poor quality; the resulting review may contain unknown biases. This paper demonstrates one way of making the effects of any discarding explicit by: (a) developing operational criteria for quality, (b) applying the criteria to 140 case studies on technological innovations in local services, and (c) indicating the differences between higher and lower quality cases. The results showed that the discarding of lower quality cases would have affected the universe of cases (higher quality cases focused more on hardware innovations, public works and transportation services, efforts supported by federal funds, larger sized efforts in terms of dollar support, and efforts with client participation), but not the overall outcomes of the innovative experience (no relationship was found between quality and service improvement or incorporation).