How Many Students, or How Many Classrooms? Design Effects for Item Analytics in Clustered Learning Platforms

Learning platforms calibrate item difficulty and discrimination directly from response logs and published minimum-sample rules tell developers how many responses an item needs. Those rules assume independent responses, yet platform responses arrive in intact classrooms. Across two structurally different platforms – a diagnostic-question service (15.9 million responses, 27,613 mathematics items, 11,838 classes) and an intelligent tutoring system – I measure the intraclass correlation of item correctness within classrooms and its design effect. The effect is estimand-specific: for item difficulty the design effect is about 2.9–4.1 (median classroom intraclass correlation 0.13 and 0.25), so the effective sample is a third or less of the nominal count, but for item discrimination it is only about 1.4. Crucially, a nominal 95% confidence interval for difficulty covers the truth only about 71% of the time and drifts slightly lower, not higher, as responses accumulate, while the textbook design-effect formula itself overstates the variance by roughly a fifth. An i.i.d. baseline covers near 95%, so the shortfall is the clustering, not the interval form. I supply corrected sample-size planners.