condagree: An R Package for Ability-Conditioned Inter-Rater Agreement in Constructed-Response Scoring

This is the authors' abstract. We don't add key points for this paper.

Inter-rater agreement for constructed-response items is usually reported as one index per item, which cannot show whether scoring is less consistent for some examinees than for others. condagree is an R package that estimates agreement as a function of examinee ability and uses a latent-category rater error model to separate a score-opportunity effect from a residual ability effect. In simulations, conditioning on the operational score rejected a true null hypothesis in up to 68% of replications, whereas the latent-model test rejected in at most 8.5%. In 14,708 double-scored PIRLS 2011 responses from Indonesia, kappa fell from .97 to .88 across achievement quintiles, a gradient explained mainly by score opportunity.