A Quality-Sensitive Method for Learning from Crowds

In real-world applications, the oracle who can label all instances correctly may not exist or may be too expensive to acquire. Alternatively, crowdsourcing provides an easy way to get labels at a low cost from multiple non-expert annotators. During the past few years, much attention has been paid to learning from such crowdsourcing data, namely Learning from Crowds (LFC). Despite their proper statistical foundations, the existing methods for LFC still suffer from several disadvantages, such as needing prior knowledge to select the expertise model to represent the behavior of annotators, involving non-convex optimization problems, or restricting the classifier type being used. This paper addresses LFC from a quality-sensitive perspective and presents a novel framework named QS-LFC. Through reformulating the original LFC problem as a quality-sensitive learning problem, the above-mentioned disadvantages of existing methods can be avoided. Further, a support vector machine (SVM) implementation of QS-LFC is proposed. Experimental results on both synthetic and real-world data sets demonstrate that QS-LFC can achieve better generalization performance and is more robust to the noisy labels, than the existing methods.