Comment on Bartko's "On various intraclass correlation reliability coefficients."

The implications of generalizabil ity theory for the choice of an intraclass correlation coefficient as a measure of the generalizability of rating data are reviewed. The review suggests that the conclusions expressed in Bartko's article on the appropriateness of certain coefficients are not entirely warranted. Furthermore, citation of the classical .test theory literature shows that his conclusion about the Kuder-Richardson Formula 20 is not correct. Bartko (1976) compared four intraclass correlation coefficients that may be used as reliability coefficients for rating data and concluded that a one-way analysis of variance (ANOVA) intraclass correlation coefficient, discussed by Bartko (1966), provides the best formulation for the reliability of ratings. The conclusion was based on a demonstration that the advocated coefficient is the only one of the four that is large if and only if the within-subjects variance is small, a characteristic he claimed is desirable in a reliability coefficient. Bartko (1976) also reported the values of two intraclass correlation coefficients derived from a two-way ANOVA of dichotomous data originally reported by Winer (1971). The value of one coefficient, which was equal to the value obtained by applying the Kuder-Richardson Formula 20 (KR-20), was larger than the value of the coefficient reported by Bartko (1966). Based on this demonstration he concluded that if for some reason the computation of an intraclass correlation from a two-way ANOVA was desirable, the Bartko (1966) approach should be used. Further, he concluded that KR-20 produces spuriously high intraclass correlations. (Bartko, 1976, p. 764.)