Large Language Models for Banking Supervision: Reliability Evidence from European Systemic Banks

The rapid expansion of narrative disclosures by systemically important banks strains supervisory capacity to extract comparable qualitative information at scale. We provide, to our knowledge, the first systematic reliability assessment of large language models for structured coding of banking disclosures, applying Claude Sonnet 4.6 and GPT-4.1 to 280 annual reports from 28 European O-SII banks over 2015–2024 across four supervisory-relevant dimensions. Intra-model reliability is excellent for Claude across all dimensions (ICC 0.856–0.974) but poor for GPT-4.1 in operational efficiency (ICC=0.487) and capital adequacy subcomponents. Inter-model consensus varies systematically with regulatory anchoring — high for climate commitments, where mandatory disclosure frameworks provide precise scoring benchmarks (r=0.907), and substantially lower for capital adequacy (r=0.375), where no equivalent framework exists. GPT-4.1 exhibits a consistent upward bias of 1.36 to 2.85 points relative to Claude across dimensions. These findings imply that model choice is not neutral for supervisory conclusions: deploying models without prior reliability assessment risks introducing systematic and undetectable errors into qualitative supervisory workflows.