Key points are not available for this paper at this time.
OBJECTIVE: The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews). METHODS: Based on a published overview investigating the hospital volume-outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five-point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated. RESULTS: The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter-rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers. CONCLUSION: The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items.
Pieper et al. (Thu,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: