Meeting the machines half-way: Testing LLMs for qualitative coding of values in texts

This article reports a small-scale experiment comparing human and Large Language Model (LLM) performance on a difficult qualitative coding task: assigning values to short text snippets. When annotators compared human and model outputs, they consistently preferred the human annotations. However, agreement among human coders was low. To explain this contradiction, we conducted follow-up interviews inspired by Cicourel’s work. The interviews revealed that human coders drew on linguistic nuance, cultural experience, and implicit assumptions about values and about the experiment. These assumptions differed from those underlying LLM outputs. We argue this is not simply a failure of LLMs. Rather, assigning predetermined categories to decontextualized text is inherently difficult for both humans and machines. Human annotators already ‘meet the machines halfway’ by being consistent and formal. Rather than treating LLMs as annotation machines, we propose three collaborative roles for LLMs: generating explanations, retrieving examples from data, and assisting with codebook development.