External Validity Bias of Purposive and Random Site Selection When Sites Can Opt Out: Evidence from the Head Start Program

Randomized controlled trials (RCTs) produce impact evidence with high internal validity but uncertain external validity. External validity bias has been defined as the expected difference between the average impact in the sample and the average impact in the population. This study estimated the external validity bias from several site selection methods by simulating hypothetical RCTs of the Head Start program in which some selected sites decline to participate. Three main findings emerged from the analysis. First, purposive site selection consistently produced biased impact estimates that varied in magnitude based on the outcome examined, the factors used to select sites, and the factors that influenced site decisions to participate. Second, simple random site selection yielded less external validity bias than purposive site selection under most tested conditions. Third, stratified random site selection yielded virtually no external validity bias, but the results likely overstate the method’s performance when data on impact moderators are unavailable before sites are selected. These findings offer lessons on how to select sites in future RCTs to minimize external validity bias.