Coding latent concepts: a human and LLM-coordinated content analysis procedure

Measuring complex and latent concepts at a large-scale poses significant challenges for communication researchers. While computational and crowdsourced methods offer solutions, they often require high professional thresholds or incur significant costs. The recent advent of large language models has revolutionized content analysis. This paper employs a human and LLM-coordinated analysis procedure to measure complex and latent concepts in 1,000 public comments, exemplified by the multi-dimension concept of “deliberativeness.” We showcase the collaboration between humans and LLMs in completing complex coding tasks by designing and refining a codebook for human use and corresponding prompts for LLMs. Surprisingly, we find that fine-tuned GPT-3.5-turbo-1106 with smaller datasets can surpass GPT-4o-2024-05-13’s performance and match manual content analyses. This paper provides communication researchers with an efficient and cost-effective reference for measuring latent concepts.