Best practices and ethical considerations for crowd-sourced data in the behavioral sciences

Many researchers in the psychological and behavioral sciences use crowd-sourcing platforms to collect data, and yet many of those researchers are poorly informed about the participant-experience on these platforms. In this paper we summarize our experience in participating in many hundreds of research tasks. We present a list of common ethical and practical short-comings perpetrated by researchers, as well as a corresponding list of recommendations for designing research that facilitates quality data, maintains ethical standards, and improves the quality-of-life of participants. We also identify how the validity of many common and standardized methods are threatened by the presence of non-naive survey-takers. Importantly, this paper includes considerable feedback and guidance from the microworker community. While there is good research being executed on crowd-sourcing platforms, we contend that most research on these platforms could be improved by adopting a subset of the recommendations herein.