Making social science reproducible: A Conceptualization of “reproducibility review” and learnings from its implementation

In reaction to the reproducibility crisis, principles of open science have gained ground in academic publishing. One emerging and promising practice is the evaluation of the data and code underlying a publication, conducted by the journal or a trusted third-party entity – reproducibility review. This article explores the adoption of reproducibility review policies across the social sciences, both conceptually and empirically. I make several contributions: First, I conceptualize reproducibility review, arguing that it can be defined along three dimensions, namely, the requirements regarding data and code sharing, the scope of the review, and its timing. Second, I survey the policies of the top journals (n = 134) across the social sciences, finding that only a minority conduct reproducibility review. Third, I provide empirical insights from two years of reproducibility review at Political Communication (n of papers = 49), for which I was responsible as the reproducibility editor. I describe required inputs (e.g., time needed) and the number of errors caught and explain these outcomes with a range of contextual variables as well as some qualitative observations.