Toward an Integration of Theory and Method for Criterion‐referenced Tests1,2

In this paper, an attempt has been made to synthesize some of the current thinking in the area of criterion‐referenced testing as well as to provide the beginning of an integration of theory and method for such testing. Since criterion‐referenced testing is viewed from a decision‐theoretic point of view, approaches to reliability and validity estimation consistent with this philosophy are suggested. Also, to improve the decision‐making accuracy of criterion‐referenced tests, a Bayesian procedure for estimating true mastery scores has been proposed. This Bayesian procedure uses information about other members of a student's group (collateral information), but the resulting estimation is still criterion referenced rather than norm referenced in that the student is compared to a standard rather than to other students. In theory, the Bayesian procedure increases the “effective length” of the test by improving the reliability, the validity, and more importantly, the decision‐making accuracy of the criterion‐referenced test scores.