In 2015, the UK Academy of Medical Sciences published a landmark report entitled ‘Reproducibility and reliability of biomedical research’ (Academy of Medical Sciences, 2015). The report brought together the perspectives of scientists, funders, journals and learned societies in tackling the high rate of irreproducibility in basic and applied biomedical science. Among the identified causes of irreproducibility were various forms of researcher bias (including p-hacking and hindsight bias), publication bias, low statistical power, underspecified methods, weak experimental designs, and uncorrected technical errors in published papers. Of the several solutions proposed, pre-registration of study procedures and analysis plans was the one strategy that singularly addressed the most problems. The case for pre-registration of hypothesis-driven research is far from new. As far back as the 1950s, Adriaan de Groot argued that pre-registration was crucial for distinguishing confirmatory hypothesis testing from more exploratory forms of analysis (de Groot, 2014), and pre-registration of clinical trials is now standard practice for controlling bias in medical research. In basic science, however, study pre-registration has generally been resisted or ignored. This ambivalence is characteristic of the now recognised ‘incentive misalignment problem’ in which practices that are important for scientific progress as a whole (including transparency and reproducibility) are out of step with those that bring career success for individual scientists (Nosek et al., 2012). The Registered Reports initiative, now offered at over 40 journals including the European Journal of Neuroscience, attempts to realign these incentives by accepting articles in advance where authors elect to pre-register their study designs (for a full list of participating journals see https://cos.io/rr/). Unlike conventional submissions, Registered Reports are peer-reviewed in two stages, both before and after results are known. Authors first submit an Introduction and Method – before undertaking their research – with the review process assessing the importance of the research question, validity of the hypotheses, and rigour of the proposed methodology. Submissions that satisfy specified review criteria are then provisionally accepted before the results are known. Armed with ‘in principle acceptance’, authors conduct the research and, once completed, resubmit a full manuscript that includes the original Introduction and Method plus the Results and Discussion. The completed manuscript then undergoes a second stage of review, with final acceptance guaranteed provided the authors adhered to their approved protocol, that the data passed pre-specified quality checks, and that the conclusions are sufficiently evidence based. Crucially, editorial decisions are made independently of the outcomes of hypothesis testing, and free from subjective assessments of the novelty or impact of findings. Full author submission guidelines for Registered Reports can be found at: http://onlinelibrary.wiley.com/store/10.1111/(ISSN)1460-9568/asset/homepages/Registered_Report_author_guide.pdf?v=1&s=5cddc7d3d6ff7938552942c6bf4b35fab38edec3&isAguDoi=false> For studies with a clear a priori hypothesis, the Registered Reports format has four key strengths. First, it prevents publication bias by ensuring that editorial decisions are based on the theoretical importance and methodological rigour of a study, before research outcomes are known. Second, by requiring authors to pre-register their study methods and analysis plans, it minimises common forms of research bias including p-hacking and presenting hypotheses derived from the results as a priori predictions (so-called ‘Hypothesising After Results are Known’, or HARKing; Kerr, 1998; John et al., 2012). Third, by requiring detailed specification of study methodology together with high statistical power (90%), Registered Reports promise to improve the repeatability of stated procedures as well as the replicability of obtained results (Button et al., 2013). Finally, because protocols are accepted in advance of data being collected, the format provides an incentive for researchers to conduct important replication studies and other novel, resource-intensive projects (e.g. involving multi-site consortia) – projects that would otherwise be too risky to attempt if successful publication is contingent on the results. The most commonly voiced concern about Registered Reports is that requiring authors to adhere to pre-specified experimental methods and analyses could limit procedural flexibility and analytic creativity. However, the format has clear safeguards to protect these aspects of scientific discovery. Minor deviations from approved procedures are permitted provided that the authors inform the editorial board immediately and any deviations are clearly footnoted in the final published report. And while authors are required to report the outcomes of all pre-registered analyses, they are free to conduct and report additional unregistered exploratory analyses; these are simply reported in a separate section of the Results to ensure that readers can distinguish confirmatory and exploratory outcomes. The Results sections of Registered Reports already published at other journals attest to the fact that the format welcomes exploratory analyses. For a virtual special issue of the first six Registered Reports published at Cortex, readers are directed to http://www.journals.elsevier.com/cortex/virtual-special-issues/virtual-special-issue-registered-reports Another frequent concern is the logistical delay associated with conducting peer review before experiments are undertaken. While it is true that authors must wait for in principle acceptance before conducting their research, this expenditure is offset by efficiency savings in other areas. One major saving arises from the statistics on manuscript rejection. The experience of Registered Reports at the most comparable adopting journal to European Journal of Neuroscience (the journal Cortex) is that the rejection rate of Registered Reports that pass editorial triage and proceed to in-depth Stage 1 review is just 10% (cf. 90% for regular submissions). This low rejection rate arises not from setting a lower bar for publication (quite the opposite), but because the Stage 1 review process is able to correct methodological problems in a study before they arise and could thus render a paper unpublishable. And because the study design is assessed at Stage 1 (and not re-assessed at Stage 2), authors who choose Registered Reports are therefore more likely to see their article published within the journal they initially submit to, avoiding the time cost of resubmitting rejected manuscripts successively to multiple journals. At Cortex, the Stage 1 and Stage 2 review processes take, on average, 10 weeks each, with (so far) up to two rounds of in-depth review in each case. Another criticism sometimes raised by life scientists is that Registered Reports seem more aligned with ‘one shot’ experiments, and may be unsuitable for sequential, programmatic research in which the results of one experiment lead to the next. In fact, the format explicitly includes an “incremental registration” option where authors can register and add experiments successively to their papers, without the full sequence needing to be specified at the outset. At each stage, the Stage 1 review process for the additional pre-registered experiments is fast-tracked and everything in the manuscript up to that point is fully accepted and ‘locked in’. Despite these features and benefits, Registered Reports should not be seen as an exclusive mode of publication or as a cure-all for irreproducibility in science. The format is not appropriate for work that is hypothesis-free (i.e. pure exploratory work or theory-free observations) or for methods development. Moreover, Registered Reports should not be seen as ‘better’ science, or as an attempt to replace exploratory science with confirmatory science. Registered Reports do nothing to diminish the value of existing article types or modes of discovery – they simply provide a new option that prioritizes deductive hypothesis testing, high statistical power, and the minimisation of bias. For a full list of FAQs regarding Registered Reports, see https://cos.io/rr/#faq More than 40 journals now offer Registered Reports. For the European Journal of Neuroscience, we are extending the format into new specialist domains, including systems neuroscience, molecular neuroscience, and clinical neuroscience. We are also expanding the reach of the format by welcoming submissions that propose secondary analyses of existing data sets, provided authors can certify that they have had no prior access to the data. As with other adopting journals, Registered Reports are published on the condition that authors agree to deposit the data underpinning any conclusions in a public archive, ensuring that readers can reproduce and extend any of the obtained findings. As we embark on this journey we welcome feedback from the scientific community and look forward to engaging productively with authors and reviewers.
No takes yet. Share an insight, caveat, or question.
Chambers et al. (2016) studied this question.
Synapse has enriched 5 related papers. Consider them for comparative context: