Reliability of Criterion‐referenced Tests: A Decision‐theoretic Formulation

It has been suggested that the primary purpose for criterion‐referenced testing in objective‐based instructional programs is to classify examinees into mastery states or categories on the objectives included in the test. We have proposed that the reliability of the criterion‐referenced test scores be defined in terms of the consistency of the decision‐making process across repeated administrations of the test. Specifically, reliability is defined as a measure of agreement over and above that which can be expected by chance between the decisions made about examinee mastery states in repeated test administrations for each objective measured by the criterion‐referenced test.