Human-annotated labels are crucial for training and evaluating machine learning models especially in domains requiring human judgment. However, crowdsourced labelers frequently struggle to reach consensus and differ from expert consensus. Through studies in sexist content identification, we demonstrate how this systematic misalignment persists even with large numbers of annotators and significantly impacts model performance. To address this challenge, we introduce cognitive empathy priming (CEP)—a scalable psychological intervention that enhances annotators' ability to recognize perspectives different from their own. Our results show that CEP substantially improves label quality: empathy-primed labelers demonstrate around 10-20% higher alignment with expert consensus compared to standard crowdsourcing methods, while inter-rater consistency also improves dramatically. These improvements translate directly to model performance, with Large Language Models trained on empathy-primed labels showing approximately 16% higher agreement with expert-determined labels compared to those trained on control group labels. Our sensitivity analyses confirm these results remain robust even when accounting for potential expert biases. This research provides organizations with an cost-effective solution to enhance AI training data quality, particularly in subjective domains like content moderation and bias detection.
Kwon et al. (Wed,) studied this question.