Rigour and transparency in the family of systematic reviews: The International Journal of Health Planning and Management encourages prospective protocol registration

Scientific research publications have grown at an astonishing rate overall, and for systematic reviews (SRs) in particular. A study on the epidemiology of SRs found that about 2500 SRs were published in Medline in 2004,1 and grew to about 8000 SRs a year when replicated a decade later.2 SRs combine evidence from primary research and stand at the top of the hierarchy of evidence, especially when meta-analyses are possible. SRs and other forms of knowledge synthesis (e.g., scoping reviews, rapid reviews, realist reviews, evidence maps)3 are a key source of synthesised evidence for informing the decision-making of clinicians as well as other health stakeholders (e.g., public health authorities, health planners, health managers, and policy-makers).4-9 To do so, the family of SRs help identify, characterise, integrate, and synthesise the array of primary research findings within a topic or research question that otherwise end-users cannot keep up with or combine for themselves. However, SRs are only valid if they are rigorous and transparent, to be trustable. With this regard, methodological assessments of SRs and other forms of knowledge synthesis continue to show that conduct and reporting gaps remain a commonplace, albeit improved over time.2, 10-14 For example, studies continue to find low compliance rates with the PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses15) reporting guidelines,2, 13 as well as selective reporting.10-12 In a remarkable example, a study comparing the published SRs with the prospectively registered study protocol has found that a third of SRs did not specify or change the primary outcome, that is, a major review protocol deviation that raises multiple questions on the issues of transparency.10 On the negative side, one can expect worse deviation rates for the conjunct of SRs which did not take the step towards increased transparency of prospectively registering their review protocols. Just the fact that one has no means to really seize the magnitude of the problem for the non-registered reviews is a major problem in itself - for the sake rigour, transparency, and trust in SRs. On the positive side, SRs with a prospective register for example, enable peer-reviewers and editors in identifying SR deviations from the final report to the planned procedures. Editors or reviewers can then ask authors for the due report of that deviation and its rationale, being able to appraise the reported deviations accordingly. Otherwise, there are no means for deviations, due or undue, to be traced. Of note, sometimes protocol deviations are justifiable or even desirable in the face of evolving circumstances, but here transparency and trust are paramount, that is, not omitting or concealing deviations, providing the rationale for the deviations, and providing the means for the deviations to be traced, reported, and appraised. When study protocols are prospectively registered, deviations can be identified, prevented, or addressed. Due deviation, honest error, and honest omissions can all occur and be addressed. However, scientific misconduct and lack of integrity can also be at the origin of protocol deviations. Scientific misconduct and integrity is a major scientific and ethical concern and a too prevalent behaviour,16-20 often encouraged by the need to publish or perish. For scientists, getting publications is a much valued indicator of scientific productivity for funding and career progression purposes.16, 21 In a sample of 315 surveyed scientists, 15% of the respondents admitted they had fabricated, falsified, plagiarised, or manipulated data in the past 3 years, and the fraud was more common among younger scientists working in a university hospital. On the reason why, 72% of the respondents rated publication pressure as ‘too high.’16 In a study published in the J Med Ethics, it was found that 53% of fraudulent papers were written by a first author who had written other retracted papers ('repeat offender'), which may represent a calculated effort to deceive. The prospective register of SR protocols increases transparency and provides further means for misconduct and undue deviations to be detected even before they got published, for example, changing eligibility criteria or outcomes analysed to turn otherwise null into positive and likely more appealing findings. As the benefits of prospectively registering SRs are widely recognised (e.g., for avoiding duplicative review efforts and for adding rigour and transparency), SR protocol registration is increasingly being recommended by many scientific journals, including the PLoS journals, BMJ, and BMC journals, to name a few. This recommendation follows in line with the PRISMA statement,15 which requires authors to report the registration status of theirs SRs, where the prospective protocol can be accessed and which deviations from what was planned to what was done has occurred, and why. The PRISMA-ScR, as the extension to scoping reviews, provides similar types of recommendations.22 Empirically, the development and registration of SR protocols has been associated to improvements in the methodological quality of SRs.23, 24 For building their protocols, authors can also rely on the PRISMA-P statement – the PRISMA statement for protocol papers,25 and in the best practices for developing scoping review protocols,26 which were designed for the sake of rigour, possible replication, and transparency. For prospectively registering SRs, a number of facilities are available, with the PROSPERO (the International Prospective Register of Systematic Reviews) and Open Science Framework (OSF) being two major not-for-profit registries that charge no fee for the process. Overall, both the PROSPERO and OSF require authors to submit protocol information about the planned design at least before data extraction takes place. PROSPERO assesses applications to assure that they are complete as well as within its scope, yet no peer review or quality appraisals are provided. Authors have the opportunity to perform amendments to the initially registered information but a track record of these is maintained. Once initially approved, PROSPERO registries become public and can be widely consulted, including by those planning to develop a new review, promoting collaborations and avoiding overlaps. These registries, necessarily reported in the final systematic review, are therefore accessible by those assessing any potential deviations. A study has found a 10-fold increase in PROSPERO registrations from 63 SRs per month in 2012 to 800 per month in 2017, providing a hint of its growing traction.27 However, another study found that in 2018, only 32% of a sample of SRs published in PubMed were registered in the PROSPERO, which was still a sizeable minority.28 In more recent years, the OSF has become a registration platform too, expanding on the options. In the OSF, there is for example, an open registration procedure which authors can use to register any form of study protocol. In all the cases above, a unique registration number is assigned to the submitted protocol. Finally, among other differences, the PROSPERO has some limits in scope (e.g., scoping reviews or SRs without human health outcomes), which do not apply to the OSF. Hence, researchers are not left without registration options. Also, in addition to registration, many authors now publish their review protocols in peer-review journal, with the added advantage of the peer-review process and improved clarity right at the protocol stage.29-32 Registration or publication options notwithstanding, a recent survey study shows that 44% of the authors did not register their protocols prior to publishing their SRs.33 On the reasons, many authors (45%) point to the lack knowledge of protocol importance, yet a percentage of respondents (37%) believed that other researchers could steal their ideas from protocol databases, and 5.3% reported that their SR had been stolen.33 Nonetheless, the majority (73%) of participants have agreed that protocol registries play a role in preventing unnecessary duplication of reviews and add to transparency, while 37% of participants agree that SR protocol registration should be mandatory.33 Provided that the overall value of protocol registration is widely recognised, especially for transparency, scientific journals have a role to play. In the era of abundant preprint repositoria and predatory journals which have minimal to no peer-review - let alone stringent peer-review policies and practices - it is important that trustable, peer-reviewed scientific journals at least recommend the use of the prospective publication of study protocols for the family of SRs. Yet, scientific journals may face the dilemma of merely recommending or encouraging registration or otherwise turning the registration into a mandatory requirement for peer-review. While mandatory approaches can provide a definitive input for SR protocol registration to be a common and required practice, and can be foreseen, a current encouragement or recommendation can provide some safeguards. Many authors might still not be aware of the requirements in the planning phase – which could have been performed years ago, for the case of longer reviews. Furthermore, especially while still a developing practice, some authors may at the current time have justifiable fears about the risks of doing so. Finally, otherwise valuable and honest reviews without a protocol registration may be denied publication. In this scenario, many journals which now recommend prospective registration of SRs haven't turned their recommendation or strong encouragement into a mandate or requirement yet. The International Journal of Health Planning and Management does not require SRs or others in the family of knowledge synthesis to be prospectively registered, yet the journal strongly encourages the practice of prospective registration for the sake of rigour, transparency, and trust in the SRs the journal publishes. Tiago Jesus, OTD, PhD, is supported from a grant from the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR grant number 90ARHF0003). NIDILRR is a Center within the Administration for Community Living (ACL), Department of Health and Human Services (HHS). The contents of this publication do not necessarily represent the policy of NIDILRR, ACL, or HHS, and you should not assume endorsement by the US Federal Government. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.