Selection bias risk in “low bias” (RoB 2) rated randomised control trials (Preprint)

Abstract Aim To establish the risk of selection bias in randomised control trials (RCT) that were overall rated as having “low bias” risk, according to Cochrane’s Risk of Bias, Version 2 (RoB 2) tool. Methods A systematic literature search of current systematic reviews of RCTs was conducted. From the identified reviews, RCTs with overall “high bias” and “low bias” RoB 2 risk ratings were extracted. All RCTs were statistically tested for selection bias risk. From the test results, true positive, true negative, false positive or false negative ratings were established, and the false omission rate (FOR) with 95% Confidence interval (CI) computed. Subgroup analysis was conduced by computing the negative likelihood ratio (-LR) concerning RoB 2 domain 1 ratings: Bias arising from the randomisation process. Results A total of 1070 published RCTs (median publication year: 2018; Interquartile range: 2013–2020) were identified and tested. A total of 7.61% of all RCTs, rated with RoB 2 as of overall “low bias” risk were discovered to be of high selection bias risk (FOR 7.61%; 95% CI: 6.31–9.14%). The likelihood for high selection bias risk in RCTs that were rated as of “low bias” risk in the RoB 2 domain 1 was 6% higher than the likelihood for low selection bias risk (-LR 1.06; 95%CI: 0.98–1.15). Conclusion The findings of this study raise issues about the validity of “low bias” risk ratings using Cochrane’s RoB 2 tool, as well as the validity of the results from 1070 recently published RCTs. Our results also suggest that the likelihood of a ”low-bias” risk rated body of clinical evidence being actually bias-free is low, and that generalization based on a limited, pre-specified set of appraisal criteria may not justify a high level of confidence that such evidence reflects the true treatment effect.