Advancing the Rigour and Integrity of Our Science: The Registered Reports Initiative

Like most scientific peer-reviewed journals, we at Stress and Health eagerly awaited the release of the 2014 journal citation metrics this summer. Indeed, we were pleased to see that our impact factor rose to 1.814, a 36% increase from the 2013 impact factor of 1.336. While this reflects highly on the quality of our contributing authors, the dedication of our editorial team and the expertise of our volunteer peer reviewers, many leading editors and journals are beginning to question the overemphasis placed on such metrics as well as the inadvertent consequences that may result from pressure on researchers and academicians to publish in high-impact journals (e.g. Brembs, Button, & Munafò, 2013; Chambers, 2013; Chambers, Feredoes, Muthukumaraswamy, & Etchells, 2014; Nosek, Spies, & Motyl, 2012; Westreich, 2015). The notion of ‘publish or perish’ has long been lore among researchers and academicians. Some fields (e.g. business) go so far as to codify A-level versus B-level journals and have very stringent requirements for granting tenure and/or promotion based on publishing in these journals. Indeed, there is even a software program called Publish or Perish (Harzing, 2007) that is ‘designed to help individual academics to present their case for research impact to its best advantage’, although the developer appropriately warns against its mechanistic use. The intent behind the emphasis on a track record of publishing in high-quality journals is a positive one aimed at developing excellence in scholars and producing cutting-edge, high-impact research that contributes to science and knowledge. The problem with this focus is that it may have the effect of undermining our own best efforts. Ironically, it is this intense pressure that can actually erode the quality and rigour that should be at the heart of science. In an effort to obtain tenure or promotion, academicians may feel pressure to ‘game the system’ through a variety of questionable tactics: piecemeal publication (breaking up a study into the largest possible number of unique publications, known colloquially as ‘salami slicing’ [Editorial, 2005]); HARKing (hypothesizing after results are known); post-hoc theorizing (finding the best theory to fit the observed significant results); ‘torturing the data until they confess’ (e.g. through the use of control variables or other means to achieve the desired p-value), sometimes termed p-hacking (Head, Holman, Lanfear, Kahn, & Jennions, 2015); and numerous other tactics to build a tenure-worthy vita. Although some would place the blame for this on disciplinary and university incentive systems, editors, authors and reviewers also contribute to this problem. Editors want to publish research that will have the greatest impact and make the most significant contributions to the field. Yet, does this discourage the important work of replicating and extending previous findings? Reviewers tend to seek out ‘significant’ results and look askance on null findings. Yet, does this inadvertently encourage HARKing and ‘torturing’ the data until something of significance (p < 0.05) arises? Authors feel compelled to tell a ‘good story’. Yet, does this lead authors to discard or fail to report results that are inconsistent with that story? While editors, authors and reviewers all potentially contribute to the problem, they can collectively contribute to its solution as well. Thus, in an effort to ‘do our part’ in this endeavour, we are pleased to announce the Registered Reports Initiative, a new publishing option available to submitting authors at Stress and Health. At the heart of this initiative is the idea that the quality of one's research should be judged by the merit of the research question, the a priori derivation of sound hypotheses and the rigour of methodology used to test those hypotheses, rather than the mere statistical significance of the final results. In essence, submission of a Registered Report is the equivalent to submitting a research proposal (akin to a grant or dissertation proposal) that contains the complete introduction and proposed methodology and data analysis prior to data collection and subsequent analysis. Submissions are evaluated based on the quality of the proposed research rather than the eventual results, avoiding unnecessary post-hoc theorizing, ‘cherry picking’ findings and other questionable practices. We encourage our authors to consider submitting their best ideas to Stress and Health via this new mechanism. Below, we provide additional details regarding commonly asked questions about Registered Reports; specific submission guidelines are also available on our journal website at http://bit.ly/1GDipb1. A Registered Report is an empirical article in which the introduction, methodology and proposed analyses are pre-registered with the journal and reviewed prior to the commencement of the research study. Thus, similar to a grant or thesis/dissertation proposal, the initial submission is evaluated based on the following: (1) the significance of the research question(s) and potential contribution of findings to knowledge; (2) the logic, rationale and plausibility of the proposed hypotheses; (3) the rigour of the proposed methodology and statistical analysis; and (4) the extent to which the methodology is sufficiently clear such that an independent investigator could replicate the procedures and analysis. Similar initiatives have been launched in other high-impact peer-reviewed journals (e.g. Chambers, 2013; Landis, Cortina, & Rogelberg, 2014; Simons, Holcombe, & Spellman, 2014). A Registered Reports submission generally has the same set of possible outcomes as a regular submission: (1) it could be rejected (either via editorial ‘desk’ rejection or after external peer review); (2) editors could ask for a revised resubmission based on peer reviewers' comments (e.g. to provide more methodological detail and/or a more robust power analysis); or (3) it could be ‘conditionally accepted’, again based on peer review. A conditional acceptance is premised upon the researchers carrying out the research protocol exactly as proposed and drawing appropriate conclusions about the findings. Upon receiving the final manuscript and reviewing to ensure that those conditions were met, a conditionally accepted Registered Report would proceed immediately to publication. The benefit to science at large is that the Registered Report is evaluated based on its intrinsic merits related solely to scientific contribution and methodology rather than the results found. Thus, it neutralizes a number of dubious research practices related to selective reporting of results, undisclosed analytic flexibility, weak statistical power and publication bias (Chambers, 2013). This also benefits submitting authors as their research is evaluated solely on the quality of their ideas rather than the outcome of their results. The Registered Report mechanism should provide researchers with reassurance and ‘peace of mind’ when conducting and awaiting the outcome of a research project as the research has been previously endorsed as worthy of investigation by a panel of peers regardless of the eventual outcome. Or perhaps, the reviewers and editor will provide authors with feedback as to why they think the research question or design is not worth publication, in their judgement, which may allay investment of significant time and energy into a project that may eventually be rejected. Such feedback may assist the researcher in further developing his or her project. Finally, because articles published under the Registered Report initiative will be identified as such, readers of the published article can be confident that these results accurately and comprehensively reflect the original research questions, hypotheses, methodology and analyses. Researchers can conduct additional exploratory/creative analyses, but these findings are reported in a separate section of the results and appropriately labelled, so readers know that these were post-hoc analyses. We recognize that research is a dynamic process and sometimes things do not always go as planned. Perhaps, the conditionally accepted Registered Report indicated that the sample size would be 200, but the authors were only able to recruit 180. Perhaps, the intended source of data (e.g. a particular organization) is no longer available. Should the study methodology deviate from the originally accepted research protocol, authors must contact the action editor immediately (prior to completion of data collection) to notify them of the modifications, however, minor. Depending on the extent of the deviation, two outcomes may occur. For minor deviations, the conditional acceptance may be preserved with the deviation from protocol reported in the final manuscript. For major deviations, the authors may be asked to resubmit the proposal as a new submission. Authors considering submission of a Registered Report are encouraged (but not required) to contact the editors regarding the suitability of a study under this mechanism, given that the Registered Report represents just one submission option at Stress and Health. Authors are still welcome to submit completed studies via our traditional ‘Research Article’ and ‘Short Communication’ mechanisms, and these will continue to be identified as such upon publication. Likely candidates for a Registered Report might include studies that have already been thoroughly reviewed as grant proposals or thesis/dissertation projects. In conclusion, we are excited about this new journal initiative. We believe that it offers authors the chance to have their research truly be evaluated on the basis of its merits and relieves them of any undue pressure to obtain significant results. Equally important, we believe that it is an important step to advancing the rigour and integrity of our scientific inquiry into stress and health.