Answers are grounded in this paper and cite their sources.
This study investigates how disagreement among human annotators affects the evaluation of sentiment analysis models for Hausa, a low-resource African language. Using the Hausa subset of the AfriSenti benchmark, we examine whether evaluating models against individual annotator judgments produces different conclusions from conventional evaluation using a single reference label. The analysis covers 4,562 eligible social media posts with three distinct annotator votes and compares classical machine learning baselines with multilingual transformer models, including AfriBERTa, XLM-R, and mBERT. Model performance is assessed using macro-F1 scores, empirical vote agreement, and paired uncertainty estimates, with particular attention to unanimous and contested annotations. Results show that overall model rankings remain consistent across both evaluation metrics, although small, statistically uncertain differences emerge on posts where annotators disagree. The study also identifies 64 instances where released sentiment labels differ from the majority of individual annotator votes. These findings highlight the importance of preserving annotator perspectives, documenting annotation coverage, and accounting for uncertainty when evaluating natural language processing systems. The research contributes a reproducible evaluation audit for Hausa sentiment analysis and provides insights into more transparent and disagreement-aware evaluation practices for low-resource languages.
No takes yet. Share an insight, caveat, or question.
Dauda et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: