A Binomial Trials Model for Examining the Ratings of Standard-Setting Judges

The purpose of this study is to describe a new approach for examining the quality of judgments obtained from standard-setting judges using a Binomial Trials Model (BTM; Andrich, 1978a). To explore the usefulness of the BTM, the judgments of 26 judges who took part in the standard-setting process for the Georgia High School Graduation Test in the content area of mathematics were examined. Ratings over 3 rounds of judgments were obtained for 60 multiple-choice items on the mathematics test. The data suggest that there is significant variability in the judged difficulties of these mathematics items within each round; the data also indicate that these judgmental item difficulties are quite stable over the 3 rounds. Furthermore, the data suggest that the judges vary significantly in their views of minimal competence, and that the differences among the judges increase over the 3 rounds. Suggestions for future research on the judgments of standard-setting judges and the practical implications of this study are discussed.