A Unified Approach for Assessing Agreement for Continuous and Categorical Data

This paper proposes several Concordance Correlation Coefficient (CCC) indices to measure the agreement among k raters, with each rater having multiple (m) readings from each of the n subjects for continuous and categorical data. In addition, for normal data, this paper also proposes the coverage probability (CP) and total deviation index (TDI). Those indices are used to measure intra, inter and total agreement among all raters. Intra-rater indices are used to measure the agreement among the multiple readings from the same rater. Inter-rater indices are used to measure the agreement among different raters based on the average of multiple readings. Total-rater indices are used to measure the agreement among different raters based on individual readings. In addition to the agreement, the paper also assess intra, inter, and total precision and accuracy. Through a two-way mixed model, all CCC, precision and accuracy, TDI, and CP indices are expressed as functions of variance components, and GEE method is used to obtain the estimates and perform inferences for all the functions of variance components. Each of previous proposed approaches for assessing agreement becomes one of the special case of the proposed approach. For continuous data, when m approaches ∞, the proposed estimates reduce to the agreement indices proposed by Barnhart et al. (2005 Barnhart , H. X. , Song , J. , Haber , M. J. ( 2005 ). Assessing intra, inter and total agreement with replicated readings . Statistics in Medicine 19 : 255 – 270 .Crossref, Web of Science ® , Google Scholar). When m = 1, the proposed estimate reduces to the ICC proposed by Carrasco and Jover (2003 Carrasco , J. L. , Jover , L. ( 2003 ). Estimating the generalized concordance correlation coefficient through variance components . Biometrics 59 : 849 – 858 .Crossref, PubMed, Web of Science ® , Google Scholar). When m = 1, the proposed estimate also reduces to the OCCC proposed by Lin (1989 Lin , L. I.-K. ( 1989 ). A concordance correlation coefficient to evaluate reproducibility . Biometrics 45 : 255 – 268 .Crossref, PubMed, Web of Science ® , Google Scholar), King and Chinchilli (2001a King , T. S. , Chinchilli , V. M. ( 2001a ). A generalized concordance correlation coefficient for continuous and categorical data . Statistics in Medicine 20 : 2131 – 2147 .Crossref, PubMed, Web of Science ® , Google Scholar) and Barnhart et al. (2002 Barnhart , H. X. , Haber , M. , Song , J. ( 2002 ). Overall concordance correlation coefficient for evaluating agreement among multiple observers . Biometrics 58 : 1020 – 1027 .Crossref, PubMed, Web of Science ® , Google Scholar). When m = 1 and k = 2, the proposed estimate reduces to the original CCC proposed by Lin (1989 Lin , L. I.-K. ( 1989 ). A concordance correlation coefficient to evaluate reproducibility . Biometrics 45 : 255 – 268 .Crossref, PubMed, Web of Science ® , Google Scholar). For categorical data, when k = 2 and m = 1, the proposed estimate and its associated inference reduce to the kappa for binary data and weighted kappa with squared weight for ordinal data.