Medical education assessment: a brief overview of concepts in generalizability theory

General Medical Council (GMC) in the UK has emphasized the importance of internal consistency for students' assessment scores in medical education.1 Typically Cronbach's alpha is reported by medical educators as an index of internal consistency.Medical educators mark assessment questions and then estimate statistics that quantify the consistency (and, if possible, the accuracy and appropriateness) of the assessment scores in order to improve subsequent assessments.The basic reason for doing so is the recognition that student marks are affected by various types of errors of measurement which always exist in student marks, and which reduce the accuracy of measurement.The magnitude of measurement errors is incorporated in the concept of reliability of test scores, where reliability itself quantifies the consistency of scores over replications of a measurement procedure.Therefore, medical educators need to identify and estimate sources of measurement error in order to improve students' assessment.Under the Classical Test Theory (CTT) model, the student's true score is the sum of the student's observed score and a single undifferentiated error term.Using this model, the most frequently reported estimate of reliability is Cronbach's alpha.Almost always, however, when alpha is reported, it incorporates errors associated with sampling of items, only.Accordingly, alpha does not allow us to pinpoint, isolate, and estimate the impact of different sources of measurement error associated with observed student marks.An extension of CTT called "G (Generalizability) theory" enables us to differentiate the multiple, potential sources of measurement error called "facets" (sometimes called "dimensions" in experimental design literature).For example, in an OSCE exam, a student might be observed by one of a large sample of examiners, for one of a large sample of standardized patients (SPs), and for one of a large sample of cases.The facets, then, would be examiners, SPs and cases---each of which serves as a potential source of measurement error.The set of all facets constitutes the universe of admissible observations (UAO) in the terminology of G theory.As another example, suppose that for a cardiology exam, the