Enhancing Bone Metastasis CT Report Analysis: A Comparison of Local and Proprietary Large Language Models for Privacy and Resource Efficiency

Abstract Background: Patients with metastatic bone tumors often suffer pathological fractures. CT reports contain critical diagnostic information, but their free-text nature makes information extraction inefficient, increases physicians’ documentation burden, and raises privacy risks. Methods: This study analyzed 300 de-identified CT reports of metastatic bone tumors and compared seven large language models (LLMs), including two proprietary models (GPT-4o, Gemini Pro) and five open-source locally deployable models (DeepSeek-V3, Qwen2-72B, LLaMA3.1-70B, Gemma2-9B, WiNGPT2-9B). Tasks included identifying pathological fractures, extracting fracture sites, and extracting metastatic sites. Expert annotations served as the reference standard. Evaluation metrics included accuracy, recall, precision, F1 score, AUC, and Cohen’s Kappa. Results: All models achieved accuracies above 80%. Open-source models such as Qwen2-72B and WiNGPT2-9B demonstrated comparable performance to GPT-4o (\((p > 0.05)\)) while offering advantages in privacy protection and local deployment. Domain-specific fine-tuning further improved the performance of smaller models (e.g., WiNGPT2-9B) in resource-constrained environments. Conclusion: Locally deployable and fine-tuned open-source LLMs can assist physicians by rapidly extracting key findings from CT reports under strict privacy protection. They reduce documentation burden, enhance diagnostic efficiency, and provide a sustainable AI pathway for healthcare systems, especially in privacy-sensitive or resource-limited settings.