Selecting Districts and Schools for Impact Studies in Education: A Simulation Study of Different Strategies

Experimental studies of educational interventions are rarely based on representative samples of the target population. This simulation study tests two formal sampling strategies for selecting districts and schools from within strata when they may not agree to participate if selected: (1) balanced selection of the most typical district or school within each stratum; and (2) random selection. We compared the generalizability of the resulting impact estimates, both to each other and to a stylized approach to purposive selection (the typical approach for experimental studies in education). We found that balanced and random selection of schools within randomly selected districts were the most consistent strategies in terms of generalizability, with minimal difference between the two. Separately, for random selection, we tested two strategies for replacing districts that refused to participate—random and nearest neighbor replacement. Random replacement outperformed nearest neighbor replacement in many, but not all, scenarios. Overall, the findings suggest that formal sampling strategies for selecting districts and schools for experimental studies of educational interventions can substantially improve the generalizability of their impact findings.