The Influence of Study Selection Criteria on Statistical Conclusion Validity for Reading Interventions: Evidence From Non-Independent Data Structures

Randomized control trials (RCTs) of Tier 2 interventions for students with or at risk for reading disabilities define study eligibility using cut points on screening measures. These criteria vary widely across studies and may inadvertently bias estimates of effectiveness. We conducted a simulation study to test how different eligibility criteria influence intervention effect estimates. Across 500 replications, we compared three common approaches: a population 40th percentile threshold, a sample 25th percentile threshold, and selecting the lowest four students per classroom. Under varying effect sizes and treatment structures, we compared the true effect with estimated intervention effects provided by subsamples yielded from each criteria method. Each eligibility criteria method produced a biased estimation of intervention effect, the degree to which varied across data structures and treatment types. These findings illustrate that eligibility decisions, while necessary for Tier 2 intervention research, produce biased and unreliable estimates, raising concerns for statistical conclusion validity. Researchers should consider the consequences of eligibility criteria when interpreting and comparing RCT outcomes. Future research is needed to find a solution that is both statistically valid and practically feasible.