From Manual Assessment to Human–LLM Research: The Changing Role of Expertise

Qualitative research is difficult to extend beyond small samples because interpretation requires expertise and validation. This study examines whether large language model (LLM)-assisted workflows can extend structured qualitative assessment while preserving analytical quality. Across twenty-five decentralised autonomous organisations and seven ordinal governance variables, assessments from four human experts were compared with a source-constrained LLM pipeline and an open-retrieval direct-prompt condition. Agreement, rating error, evidence coverage, stability, and auditability were evaluated separately. Because the LLM conditions differed simultaneously in model, retrieval, interface, and aggregation, they are treated as compound conditions rather than as a causal workflow comparison. The results support a conditional conclusion. LLM performance depended on the assessed variable, available evidence, and model behaviour. For several interpretative constructs, individual LLM–human agreement was similar in magnitude to agreement among human experts. However, broader coverage, granular provenance, and stable outputs did not necessarily coincide. The study therefore demonstrates methodological feasibility rather than realised scaling. LLMs can extend structured qualitative research, including the assessment of complex organisational constructs such as organisational autonomy, but their outputs are not self-validating. Human expertise therefore remains essential, although its role shifts from conducting every assessment manually towards defining constructs and evidence rules, validating model and workflow performance, monitoring bias and abstention behaviour, and adjudicating disagreement. A detailed methodological understanding of the selected model and workflow is consequently integral to rigorous qualitative research. Preprint, not peer reviewed. The accompanying Supplementary Methods contain Appendices A–D. Supporting data, code and research materials: https://doi.org/10.5281/zenodo.23480402.