Efficient Task Assignment for Multi-Workerset Crowdsourcing with Time and Expense Considerations

Abstract Large-scale crowdsourcing with high-quality results, such as online surveys and data labeling tasks, is in wide demand. Existing techniques on task assignment result quality optimization, however, have addressed only a part of optimization space that does not necessarily reflect the real-world problem. For example, in reality, the proper worker set for task assignment and the workers’ submissions to the task are non-obvious in advance, which is not considered by existing techniques. Hence, this paper discusses a task assignment algorithm that dynamically probes the proper worker set(s) and worker submission models for the task to optimize task assignments in terms of quality, time, and expense. Specifically, we introduce CrowdBwO (Crowd Bandit with Optimization), a novel multi-armed bandit algorithm that is based on batched bandits and bandits with Knapsack and incorporates worker submission models. CrowdBwO dynamically determines and utilizes proper worker set(s) and worker submission models for each task under uncertainty to achieve high-performance crowdsourcing. We conducted extensive experiments with synthetic workers and real workers to evaluate CrowdBwO in two specific problem settings. Our extensive experimental results demonstrate that CrowdBwO is significant for real-world crowdsourcing and has a high performance.