Hypothetical nudges provide directional but noisy estimates of real behavior change

Hypothetical scenarios provide an extremely useful alternative to field experiments for scholars interested in nudging behavior change, comprising a substantial proportion of the literature. Yet the extent to which hypotheticals accurately estimate real-world treatment effects is not well understood. To investigate, we identified five recent field studies of real-world nudges in distinct domains and designed four styles of hypothetical scenarios to approximate each one. This setup allows for clear comparison of old field data with new hypothetical data. Across our 20 experiments (N=16,114), hypothetical scenarios nearly always estimated the correct direction of treatment effects. However, they varied widely in estimating magnitudes, making them unreliable inputs to real-world policy applications such as cost-benefit analyses. Our findings underscore the promising value of hypotheticals, but also the need for greater investigation into strategies to calibrate their estimates.