Models and Crowdsourcing Prompts for the Automatic Detection of Malicious Worker Behavior in Crowdsourced Paraphrases

Automated paraphrasing has numerous applications such as query-answering, dialog systems, and machine translation. Recently, due to the limitations of automatic paraphrasing systems, crowdsourcing emerged as an alternative for obtaining corpora of paraphrases. Since crowdsourced paraphrases are generated by crowd workers with varied skills and motivations, such paraphrases may suffer from various quality issues (e.g. grammatical and spelling errors). Such quality issues stem from the lack of quality assessment methods for open-ended text in crowdsourcing platforms. Thus unqualified or malicious workers may generate erroneous paraphrases. Crowdsourced paraphrases must be assessed to filter low quality paraphrases, imposing an extra cost. Detecting such erroneous paraphrases requires an understanding of how malicious workers generate paraphrases. In this paper, we propose an effective approach to detect malicious worker behaviors based on a comprehensive analysis of cheating behavior characteristics. We investigate what models and features are effective in detecting such behaviors, and explore mechanisms and crowdsourcing prompts that could lead to an effective integration in an actual crowdsourcing task. Our results point to a significant reduction of cheating behavior (from 18% to 4.4%) in applying automatic malicious worker behavior detection, and highlight the impact of the detection (and detection mistakes) on workers corrective measures.