Generalizability Theory and Classical Test Theory

Abstract Broadly conceived, reliability involves quantifying the consistencies and inconsistencies in observed scores. Generalizability theory, or G theory, is particularly well suited to addressing such matters in that it enables an investigator to quantify and distinguish the sources of inconsistencies in observed scores that arise, or could arise, over replications of a measurement procedure. Classical test theory is an historical predecessor to G theory and, as such, it is sometimes called a parent of G theory. Important characteristics of both theories are considered in this article, but primary emphasis is placed on G theory. In addition, the two theories are briefly compared with item response theory. Notes An earlier version of this paper was presented at the 2008 annual meeting of the American Educational Research Association. The paper was one of two presented in a symposium sponsored by the Buros Center for Testing, the sponsor of this journal. The other paper enumerated the benefits of item response theory. We hope to be able to present this item response theory paper in a future issue of the journal. 1For more complete overviews of CTT see CitationLord and Novick (1968), CitationFeldt and Brennan (1989), and CitationHaertel (2006). For more complete overviews of G theory see CitationCronbach, Gleser, Nanda, and Rajaratnam (1972) and CitationBrennan (1992, Citation2001b). 2Equivalently, for any indefinitely large subpopulation of examinees, the expected value of the errors is 0 provided examinees are not selected based on their observed scores. 3One complexity is that reliability coefficients have nonlinear characteristics. That is why it is much more difficult to raise a reliability coefficient from .90 to .95 than from .50 to .55. 4Classically parallel forms satisfy the assumptions of essential tau-equivalence, but this is not necessarily true for congeneric forms. 5The more familiar estimation formula for Lord's SEM in the mean-score metric is: 6It need not be true that n′ t = n t nor that n′ r = n r ; that is, the sample sizes used to estimate variance components need not equal the sample sizes used in an operational form of the test. 7D study designs can differ with respect to structure and/or sample sizes. 8Strictly speaking, for a random facet it is assumed that the number of conditions in the universe of generalization is indefinitely large. 9That is, the D study design shall be p × T × R with n′ t prompts and n′ r raters. 10The most common "trivial" case is a design and universe with a single random facet. 11CTT deals with sample size changes through the Spearman-Brown formula (see, CitationFeldt it merely masks it. 13It can be argued that stratified alpha (CitationCronbach, Schönemann, & McKie, 1965) is a CTT precursor to multivariate G theory. 14A mixed-model univariate analysis effectively makes a statistical "hidden" choice for the w weights for each fixed level, whereas a multivariate analysis leaves the choice of weights to the investigator. 15If items are considered as congeneric forms, then perhaps this problem can be circumvented (L. S. Feldt, personal communication, March 3, 2010). 16It might be argued that ENR is an expected value over a propensity distribution of performance on the fixed items, but even then, the items (or item parameters) themselves are still fixed. 17Bayesian priors are actually involved in the CitationBriggs and Wilson (2007) and CitationChien (2008) approaches, which employ MCMC methods.