Answers are grounded in this paper and cite their sources.
Registry-based analysis reveals high inter-LLM agreement but substantial divergence from official NACE codes across 43,931 firms, indicating taxonomic ambiguity drives classification gaps.
This study examines whether patterns of divergence between large language model (LLM) firm classifications and the Statistical Classification of Economic Activities in the European Community (NACE) are more consistent with between-model inconsistency, information-availability effects, or taxonomic ambiguity. Using a registry dataset of 43,931 Finnish firms, of which 29,565 received at least one Experience Industry classification, we compare GPT-3.5, GPT-4, Gemini 1.5 Flash, and Llama 3 70B. LLMs were prompted using company name and identifier only, and the resulting keywords were mapped to sector profiles, enabling comparison between model outputs and formal taxonomic assignments without external descriptive text. We evaluate agreement with Krippendorff’s α, cosine similarity, and normalized HHI-based concentration measures, and examine variation by firm revenue, size, and sector-level keyword reliability. Using raw LLM-derived profiles, inter-LLM similarity is high (cosine 0.89–0.95), whereas LLM–NACE alignment is substantially lower (0.19–0.24). Confidence-level adjustment increases LLM–NACE similarity, as expected, because adjusted profiles partly incorporate NACE information. Agreement increases in revenue-weighted and larger-firm analyses, but declines in sectors with low keyword reliability and diffuse boundaries. Across all weighting schemes and size classes, inter-LLM similarity exceeds LLM–NACE similarity. The findings suggest that LLM–NACE divergence is more consistent with information-availability effects and taxonomic ambiguity than with large between-model inconsistency. The study contributes a multi-metric framework for evaluating classification under unstable category systems and shows how LLM-based profiling can complement formal industry coding in hybrid sectors.
No takes yet. Share an insight, caveat, or question.
Mika Naumanen (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: