Evaluating evidence and making judgements of study quality: Loss of evidence and risks to policy and practice decisions

Attempts to evaluate the effectiveness of health promotion interventions require judgements to be made about the quality of the available evidence. While recognizing the need to evaluate the quality of evidence, this paper argues that conclusions about effectiveness are made on the basis of too little information with too much evidence discarded. This argument is supported by a brief review of 88 health promotion systematic reviews showing large swathes of the available evidence failing to feature in reviews. Unfortunately, there is a lack of consistency in the criteria used to make decisions about whether evidence is acceptable or not and furthermore no external basis exists for validating decisions. Three approaches to evaluating evidence are considered which result in different decisions being made about individual studies. It is concluded that the evaluation of evidence cannot be reduced to the adherence to a set of simple rules and that the process requires judgement and balancing of a number of different considerations. Furthermore, evaluations of programme effectiveness should be based on a consideration of all of the available evidence and not just on a selected sub-set of studies.