Bivariate Coefficients of Agreement among any Number of Observers

The community of preference of N judges, or raters, can be quantified with coefficients measuring the agreement of the values assigned by the judges to a set of H units ("classes"). In this paper, new coefficients of agreement are suggested for the measure of intraclass consistency between observations on two variables. The coefficients are derived from a general coefficient for measuring intraclass dependence in a bivariate analysis context. It is shown that various coefficients for the univariate agreement analysis (Krippendorff's r; Fisher's intraclass correlation coefficient; Cohen's k; Scott's π Robinson's A, and Kendall's W) are particular cases of the suggested coefficients.