The Risks of Using Large Language Models for Text Annotation in Social Science Research

Generative artificial intelligence (AI) or large language models (LLMs) have revolutionized computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks of using LLMs for diverse coding tasks in social movement studies. We propose a framework for social scientists to adopt LLMs to text annotation, either as the primary coding decision-maker or as a coding assistant. Additionally, we discuss the associated epistemic risks related to validity, reliability, replicability, and transparency. We conclude by offering several practical guidelines for using LLMs in coding tasks.