Introducing Synchronous Robustness Reports: Guidelines for Journals

The vast majority of empirical research articles feature a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team. However, recent multi-analyst projects have demonstrated that different analysis teams usually adopt a unique approach and that there exists considerable variability in the associated conclusions. There appears to be no single optimal statistical analysis plan, and different plausible plans need not lead to the same conclusion. A high variability in outcomes signals that the conclusions are relatively fragile and dependent on the specifics of the analysis plan. Crucially, without multiple teams analyzing the data, it is difficult to gauge the extent to which the conclusions are robust. We propose that empirical articles of particular scientific interest or societal importance are accompanied by two or three short reports that summarize the results of alternative analyses conducted by independent experts. The present paper aims to facilitate the adoption of this approach by providing concrete guidance on how such Synchronous Robustness Reports could be seamlessly integrated within the present publication system.