Challenges with Label Quality for Supervised Learning

Organizations that develop and use technologies around information retrieval, machine learning, recommender systems, and natural language processing depend on labels for engineering and experimentation. These labels, usually gathered via human computation, are used in machine-learned models for prediction and evaluation purposes. In such scenarios, collecting high-quality labels is a very important part of the overall process. We elaborate on these challenges and discuss research directions.