Unmasking survey fraud: investigating data quality issues in an MTurk sample

Social scientists increasingly rely on Amazon Mechanical Turk (MTurk) for survey participant recruitment, but emerging research suggests a decline in data quality, raising concerns about its reliability. In November 2023, a sample of 221 U.S. MTurk workers was recruited for a survey experiment examining the impact of affordable housing rhetoric on self-esteem. Despite implementing several best practices for recruitment on MTurk – such as system qualifications, screening questions, and virtual private server/network detection – we found that an estimated 65–84% of the workers seeking compensation submitted fraudulent survey responses. This study details five strategies we used to identify fraudulent data: survey re-entry, duplicate demographic data, similar open-ended responses, nonsensical open-ended responses, and repeated geographic coordinates. Our findings reveal critical shortcomings in current MTurk best practices and suggest that, without additional third-party fraud prevention tools or thorough data screening procedures, the platform may no longer be suitable for rigorous academic research.