Modeling Agreement among Raters

Abstract An approach to the modeling of agreement among raters is proposed. By examining a hierarchy of log-linear models, it is shown how one can analyze the agreement among the raters in a manner analogous to the analysis of association in a contingency table. Specific attention is given to the problems of the K-rater agreement and the agreement between several observers and a standard. Examples are used to illustrate how this approach provides a general framework for modeling agreement in a variety of problem situations.