Assessing and Testing Interrater Agreement on a Single Target Using Multi-Item Rating Scales

An index for assessing interrater agreement with respect to a single target using a multi-item rating scale is developed. The variance of rater mean scale scores is used as the numerator of the agreement index. Four variants of a disattenuated agreement index that vary in the random response term used as the denominator are examined. A variant of rwg(J)using perfectly collinear, uniformly distributed responses as the baseline is recommended as a useful index of disattenuated agreement. A test of the equality of variances is recommended for deleting items with a level of interrater agreement significantly different from zero but lower than that of the remaining items.