Automated large language model screening of unstructured radiology reports for timely osteoporosis intervention

To develop a large language model (LLM) based approach to classification of vertebral osteoporotic compression fractures (VOCFs) from unstructured lumbar spine radiographs, for integration into osteoporosis clinical workflow. We evaluated the performance of two proprietary LLM tools using a few-shot learning approach in classification of VOCFs from unstructured radiology reports into five different subcategories of fracture as well as an overall classification of “recent fracture for notification”. Evaluation was conducted on two real-world datasets authored by 100 unique radiologists. Model outputs were benchmarked against a human-reference standard established by radiologists. Both LLM tools demonstrated satisfactory performance with sensitivity scores above 0.70 and specificity score above 0.80 for all datasets. One LLM tool achieved higher sensitivity scores which aligns with the intended role as a screening tool for osteoporotic fractures. There was very high agreement between the 5 runs of each model (κ > 0.81) as well as consistency between datasets of different timepoints (Model 1, χ² = 2.46, p = 0.12; Model 2, χ² =1.50, p = 0.22). The most common LLM errors were related to temporal context confusion and complex syntax. Both LLM tools demonstrated satisfactory performance without any fine-tuning in identification of recent osteoporotic fractures from unstructured lumbar spine radiographs. Our study demonstrates the feasibility of an LLM-based approach to classify and flag radiology reports for early intervention by relevant clinical partners, particularly, in this study, to the osteoporosis unit for expedited screening and management.