Claims about “a hidden universe of uncertainty” in social research are exaggerated: How many-analyst studies risk overestimating uncertainty

We reassess the finding of a widely discussed many-analyst study (Breznau et al. 2022) that analysts' results vary widely even when working with the same data on the same social science research question (that immigration reduces support for social policies). Only a small fraction of the variance in the estimates (5 %) could be explained by visible methodological choices. This is troubling because it implies that social research has little credibility. Moreover, the hidden nature of uncertainty would make it impossible to suggest ways to improve research. In our reanalysis, we found several shortcomings in the meta-analysis of the data, including rescaling errors and mis-specified meta-regressions. Using a more efficient meta-regression approach, we can explain most of the variance in the estimates (64 %). We conclude that the study vastly overstated the uncertainty of social research. We provide several recommendations on how future many-analyst studies could avoid misleading results.