On the Marginal Dependency of Cohen’s κ

Cohen’s κ (kappa) is typically used as a measure of degree of rater agreement. It is often criticized because it is marginal-dependent. In this article, this characteristic is explained and illustrated in the context of (1) nonuniform marginal probability distributions, (2) odds ratios that remain constant while κ changes in the presence of varying marginal distributions, and (3) percentages of raw agreement that remain constant while κ changes in the presence of varying marginal distributions. The meaning and interpretation of κ are explained with reference to the log-linear main effect model of variable independence. This model is used for the estimation of the expected cell frequencies of agreement tables. It is shown that the interpretation of κ as a measure of degree of agreement is incorrect. The correct interpretation is that κ assesses the degree of agreement beyond that expected based on a statistical model such as the independence or the null model. Based on Goodman’s (1991) distinction between marginal-free and marginal-dependent measures, it is shown that κ is marginal-dependent. It shares this characteristic with the well-known χ 2 -statistic and the correlation coefficient for cross-classifications. In contrast, the odds ratio, the unweighted log-linear interaction, and the percentage of raw agreement are marginal-free. Therefore, the expectation that marginal-dependent κ would reflect the same data characteristics as some of the marginal-free measures is misguided. It is recommended that researchers report both measures of degree of agreement and measures of agreement beyond some expectation.