Automated ICD-O-3 Coding of Real-World Pathology Reports Using Self-Hosted Large Language Models

While large language models (LLMs) have shown promise in medical text processing, their real-world application in self-hosted clinical settings remains underexplored. Here, we evaluated five state-of-the-art self-hosted LLMs for automated assignment of International Classification of Diseases for Oncology (ICD-O-3) codes using 21,364 real-world pathology reports from a large German hospital. For exact topography code prediction, Qwen3-235B-A22B achieved the highest performance (micro-average F1: 71.6%), while Llama-3.3-70B-Instruct performed best score at predicting the first three characters (micro-average F1: 84.6%). For morphology codes, DeepSeek-R1-Distill-Llama-70B outperformed other models (exact micro-average F1: 34.7%; first three characters micro-average F1: 77.8%). Large disparities between micro- and macro-average F1-scores indicated poor generalization to rare conditions. Although LLMs demonstrate promising capabilities as support systems for expert-guided pathology coding, their performance is not yet sufficient for fully automated, unsupervised use in routine clinical workflows.