On the statistical analysis of experiments with manipulation checks

Psychologists are often interested in the effect of an internal state, such as ego depletion, that cannot be directly assigned in an experiment. Instead, they assign participants to a manipulation intended to produce this state, using manipulation checks to assess the manipulation's effectiveness. We discuss statistical analyses for experiments in which researchers are primarily interested in the average treatment effect (ATE) of the target internal state rather than that of the manipulation. Often, researchers estimate the association of the manipulation itself with the dependent variable, but this intention-to-treat estimator is typically biased for the ATE of the target state, and the bias could be either toward the null (conservative) or away from the null. We discuss the fairly stringent assumptions under which this estimator is conservative. Given this, we argue against the status-quo practice of interpreting the ITT estimate as the effect of the target state, without any explicit discussion of whether these assumptions hold. Under a somewhat weaker version of the same assumptions, one can alternatively use instrumental variables (IV) analysis to directly estimate the effect of the target state. IV analysis complements intention-to-treat analysis by directly addressing the central question of interest. As a running example, we consider a multi-site replication study on the ego-depletion effect, in which the manipulation’s partial effectiveness led to criticism and several re-analyses that arrived at varying conclusions. We use IV analysis to directly account for the manipulation’s partial effectiveness; this corroborated the replication authors’ reported null results.