Answers are grounded in this paper and cite their sources.
This study proposes to apply the Theil-index ratios for the interrater reliability. We discuss the theoretical foundations and examine its function empirically using real data. Our analyses show that Theil-index rations and intraclass correlation (ICC) estimates are highly correlated. However, ICC may underestimate the interrater reliability by some extreme disagreement among raters and be more likely to be influenced by the extreme disagreement. As Theil-index ratios overcome the limitations of ICC to some degree, it seems that Theil-index ratios provide an alternative to evaluating interrater reliability, at least under certain conditions, e.g., when outliers exist in the data, it is difficult to obtain the variance component estimates, or ICC underestimates interrater reliability.
No takes yet. Share an insight, caveat, or question.
Tianshu Pan (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: