Using Mokken scaling to evaluate rating quality with anchor designs

Nonparametric methods adapted from Mokken Scale Analysis (MSA) offer an exploratory approach to evaluating the quality of rater judgments in performance assessments that is grounded in invariance requirements. However, typical applications of MSA-based methods require complete data, where all raters score all examinees. We use a real data illustration and simulation study to demonstrate how invariant ordering analyses adapted from MSA can be used with incomplete rating designs. We focus on anchor rating designs, where all raters score a common group of examinee performances, but all other performances are scored by two randomly selected raters. Our real data demonstration uses data from a rater-mediated music performance assessment, and our simulation study includes conditions that vary with respect to sample size, anchor set characteristics, and the presence and type of rater effects. Our results support the use of the adapted MSA techniques to evaluate rating quality in performance assessment contexts with anchor rating designs. We discuss the implications of our results for research and practice.