The threat of AI chatbot responses to crowdsourced open-ended survey questions

Many researchers use crowdsourced and online surveys and open-ended survey questions to gather new ideas and concerns from the public regarding emerging energy and climate technologies. The availability of large language model chatbots, most notably ChatGPT, presents a threat to the utility of these approaches. While closed-ended questions and paradata analysis have been previously used to screen inattentive and ingenuine respondents, and open-ended question responses can be manually examined for unrelated content, these chatbots can mimic satisfactory answers to evade typical detection. Using a question asking people about their thoughts about hydrogen energy, in a side-by-side comparison of a managed panel sample ( N = 834) to one drawn from Amazon Mechanical Turk ( n = 1166), I find that responses from the latter appeared higher-quality but were more likely to originate from AI chatbots, even after screening based on close-ended questions and survey paradata. Survey designs should thus incorporate structural changes to prevent fraudulent responses, and analysis going forward must improve methods to detect them. • Perspective