Although previous research has described that intervention effects vary across replication studies, less effort has been devoted to identifying causes of this effect heterogeneity with regard to differences in study implementations. However, knowing in which way study characteristics (such as population, measurement instrument, setting, or treatment implementation) impact the study results may not only help to better infer the impact of research practices but also provide evidence for theory building. Causal effects can be easily identified if all study characteristics but the one under investigation are kept constant across two studies. This is, however, not always possible in practice and unintended differences between the studies to be compared may confound the relationship of the study characteristic of interest and the treatment effect. In this article, we present a statistical approach for identifying effects of study characteristics on study-specific treatment effects from randomized experiments in cases in which unintended differences in study implementation across studies cannot be prevented. We present formal definitions of the causal effects of interest, identification assumptions, and derive respective causal estimands. The assumptions can more likely be fulfilled in prospective replication studies or many-lab studies, where researchers have more control over design and measurement of covariates in both studies. We also provide ways to test the assumptions and illustrate consequences of not meeting the assumptions. The approach is illustrated using an empirical example on the imagined intergroup contact effect in social psychology. (PsycInfo Database Record (c) 2026 APA, all rights reserved).
Pohl et al. (Mon,) studied this question.