Assessment of selective reporting biases in studies included in Campbell Systematic Reviews: A systematic review

BackgroundCritical appraisal of the studies included in a systematic review is essential to ensure that resultsof the review are properly interpreted. Critical appraisal is also one of the most difficult steps inresearch reviews. Structured risk of bias (ROB) tools can facilitate critical appraisal, but thesetools vary in content and structure, and there are unresolved issues in applications of thesetools. Assessment of risk of reporting biases, such as outcome reporting bias (ORB) and analysisreporting bias (ARB), is especially difficult, given the lack of availability of the raw materials(such as prospectively registered protocols or analysis plans) needed to properly assess the riskof selective reporting and selective non-reporting of outcomes and analyses.ObjectivesTo identify methods used in recent Campbell systematic reviews of intervention effects toassess the risk of reporting biases in included studies.Search methodsWe searched the Campbell Library website, using a structured online form developed for thispurpose, with filters for publication dates (all dates in 2020 through April 2023) and type ofdocument (completed reviews only).Selection criteriaWe included systematic reviews (SRs) of primary studies of intervention effects published inCampbell Systematic Reviews between 1 January 2020 and 30 April 2023.Data collection and analysisOf the 59 SRs published from 2020 through early 2023, 51 were eligible for our review. Fortynineof these reviews included relevant studies of intervention effects. From these 49reviews, we extracted data on methods used to assess risk of reporting biases (ORB and ARB),broader risk of bias (ROB) or study quality assessments, and adherence to 12 mandatorymethodological standards. Data extraction and coding were performed in duplicate, by pairsof team members who worked independently, and any discrepancies were resolved bycoders or by the review team. Results were compiled in a spreadsheet, which was used togenerate tables, graphics, and a narrative summary.Main resultsReporting biases were defined and assessed in diverse and sometimes idiosyncratic ways inrecent Campbell systematic reviews of intervention effects. Most (40 of 49) reviews conductedsome structured assessment of reporting biases, but many did not report results of theseassessments. Explanation and documentation of ORB and ARB assessments was missing inmore than half (28) of the reviews. Only 12 reviews provided full documentation for theirORB/ARB assessments.Overall, we found that reviewers’ descriptions of their assessments of reporting biases wereoften incomplete and inconsistent across studies. In many cases, these assessment practices did not reflect current understanding of the prevalence of selective reporting and ways in whichthese biases can undermine the validity of and confidence in results of research reviews. Thisobservation is consistent with the fact that most reviewers did not consider the potentialimpacts of risks of bias on the credibility of their results.None of the recent reviews appeared to meet all (12) of the mandatory methodologicalstandards we assessed. On average, these reviews failed to meet 4.9 of these standards (SD =2.3); almost three-quarters (35) of the reviews failed to meet four or more standards.Authors' conclusionsRecent Campbell reviews did not consistently appraise or document risks of reporting biases inthe studies they included. Assessment of risk of reporting biases is difficult, given the lack ofavailability of prospective, public protocols or analysis plans for most studies.Reviewers’ failure to adhere to Campbell’s mandatory methodological standards and editors’apparent inability to enforce these standards can be understood as functions of the contexts inwhich systematic reviews are highly desirable, highly cited, and under-resourced.We provide a decision tree to guide reviewers’ assessments of reporting bias, along with ninerecommendations for improving these practices in systematic reviews of intervention effects.Our recommentations include: more deliberate use of eligibility criteria to eliminate studiesthat cannot provide valid answers to review questions, thorough documentation of reviewers’assessment processes and ROB ratings, and explicit use of ROB ratings in interpretation ofresults.