We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced.
No takes yet. Share an insight, caveat, or question.
Sorokin et al. (2008) studied this question.
Synapse has enriched 4 closely related papers on similar clinical questions. Consider them for comparative context: