Planning for New Threats to Online Research Data Validity: The Issue of Computer-Using Agents

Increasing numbers of research studies are completed entirely online, often using “crowdsourcing” platforms to recruit participants, such as Amazon’s Mechanical Turk (MTurk) or Prolific. Over time, crowdsourced data collection has faced multiple threats to validity including bots, inattention, and misrepresentation. Fortunately, research and development of quality control techniques has resulted in approaches to conducting credible research using crowdsourcing platforms. However, advances in large language models (LLMs), particularly computer-using agents (CUAs), such as OpenAI’s “Operator,” may pose novel threats to the validity of crowdsourced research data in the future. This brief report explores how and why that might be the case and encourages further discussion of this issue.