Extraverted introverts, cautious risk-takers, and selfless narcissists: A demonstration of why you can’t trust data collected on MTurk

Several prior studies have used advanced methodological techniques to demonstrate that there isan issue with the quality of data on Amazon’s Mechanical Turk (MTurk). The goal of the presentpreregistered project was to provide an accessible demonstration of this issue by showing thatpairs of items assessing clearly contradictory content (e.g., “I talk a lot” and “I rarely talk”) arepositively correlated on the platform. We administered 27 of these semantic antonyms to samplesdrawn from Connect (N1 = 100), Prolific (N3 = 100), and MTurk (N2 = 400; N4 = 600). Despitemost of the antonyms being negatively correlated on Connect and Prolific, over 96% of theantonyms were positively correlated on MTurk. This issue could not be remedied by screeningthe data using common attention check measures nor by recruiting only “high-quality” MTurkparticipants. These findings provide clear evidence that data collected on MTurk cannot betrusted.