Comparison of Reliability Measures Under Factor Analysis and Item Response Theory

Reliability of test scores is one of the most pervasive psychometric concepts in measurement. Reliability coefficients based on a unifactor model for continuous indicators include maximal reliability ρ and an unweighted sum score–based ω, among many others. With increasing popularity of item response theory, a parallel reliability measure π has been introduced using the information function. This article studies the relationship among the three reliability coefficients. Exploiting the equivalency between item factor analysis and the normal ogive model, π (2) for dichotomous data is shown to be always smaller than ρ. Additional results imply that ω is typically greater than π (2) in practical conditions, though mathematically there is no dominant relationship between π (2) and ω. Furtherresults indicate that, as the number of response categories increases, π can surpass ω. The reasons why π and ω fall short of ρ are also explored from an information gain/loss perspective. Implications of the findings on scale development and analysis are discussed.