The gold standard for whom? Schools' experiences participating in a randomised controlled trial

Background Randomised controlled trials (RCTs) have long been considered the gold standard in education research. Federal funds are allocated to evaluations that meet What Works Clearinghouse standards; RCT designs are required in order to meet these standards without reservations. Schools seek out interventions that are research based, in other words, interventions that have demonstrated effects in RCTs, creating a cycle that incentivises RCT research. However, RCTs pose tremendous obstacles for school‐based participants. Methods This paper presents a descriptive case study of an RCT of an adolescent reading intervention implemented in four districts over three years, drawing on interviews and document analysis to illustrate the challenges that participation in an RCT poses for schools. Results In two participating districts, the random assignment requirement caused major difficulties with recruitment and retention. Of the 15 schools that began implementation in fall of year 1, 11 left the study by fall of year 2. Of those 11, eight explicitly stated that the random assignment requirement was the reason they withdrew. Stakeholders expressed that they felt it was unethical to withhold the intervention from some students. Conclusions This rate of attrition casts doubt on the external validity of the programme evaluation, as these schools are likely to systematically differ from schools that met the requirements of random assignment. Suggestions are offered for alternative methods that may be used to evaluate promising interventions, in ways that are better aligned with real‐world educational contexts. Highlights What is already known about this topic RCTs are considered the gold standard in education research, and research funding is distributed accordingly. Promising interventions often fail to replicate under effectiveness conditions. What this paper adds This paper examines the challenges of participation in an RCT for schools. Student‐level random assignment presents a barrier to school participation in RCTs. High attrition rates cast doubt on the external validity of the findings. Implications for theory, policy and practice When designing intervention research beyond the efficacy stage, feasibility for practitioners and external validity should be given greater weight. Alternative methods for evaluating promising interventions are proposed, including switching replications designs, RCTs comparing interventions with no business as usual control group, regression discontinuity, matched non‐random comparison groups, and improvement science approaches.