Methodology for Examining the Reliability of Group Mean Difference Scores

This article treats various procedures for examining the reliability of group mean difference scores, with particular emphasis on procedures from univariate and multivariate generalizability theory. Attention is given to both traditional norm‐referenced perspectives on reliability as well as criterion‐referenced perspectives that focus on error‐tolerance ratios and functions of them. The procedures discussed are illustrated using three cohorts of data for third‐ and fourth‐grade students in Iowa who took the Iowa Tests of Basic Skills in recent years. For these data, estimates of reliability for norm‐referenced decisions tend to be relatively low. By contrast, for criterion‐referenced decisions, estimates of reliability‐like coefficients based on error‐tolerance ratios tend to be noticeably larger.