The question of data reliability is of first importance to assess the quality of manually annotated corpora. Although Cohen ' s is the prevailing reliability measure used in NLP, alternative statistics have been proposed. This paper presents an experimental study with four measures (Cohen's , Scott's , binary and weighted Krippendorff ' s ) on three tasks: emotion, opinion and coreference annotation. The reported studies investigate the factors of influence (annotator bias, category prevalence, number of coders, number of categories) that should affect reliability estimation. Results show that the use of a weighted measure restricts this influence on ordinal annotations. They suggest that weighted is the most reliable metrics for such an annotation scheme.
No takes yet. Share an insight, caveat, or question.
Antoine et al. (2014) studied this question.