Diagnosing the role of observable distribution shift in effect generalization for psychological experiments

Abstract Social science researchers are making increasing efforts to replicate important experiments, compare effect estimates and analyse their discrepancies. This article is motivated by a replication of a psychological experiment in which a decline in the effect of an eye movement intervention on false memory has been attributed to a distributional shift in participants’ depression scores. Understanding how such shifts contribute to effect discrepancy is crucial for result reporting, scientific understanding, and treatment deployment. We propose a framework using generalizability methods to decompose effect size discrepancy between two studies into contributions of various sources of distribution shifts. We address several unique challenges in the application, including incorporating commonly cited sources of shift within a unified framework, providing interpretable summaries of contributions, dealing with limited sample sizes and selection bias. In the motivating study, our analysis reveals that, despite the notable shift in observed depression scores, due to limited effect heterogeneity, their contribution to effect discrepancy is less compelling than that of unobserved factors. In two additional pairs of experiments, our method offers context-dependent insights on the contributions of different factors in effect generalization. The proposed tools are especially useful early in the scientific process when there are not enough studies for a meta-analysis.