Automated risk scoring for venous thromboembolism using large language models with expert knowledge-augmented prompting: a multicenter validation study

Venous thromboembolism (VTE) is a common but preventable complication in hospitalized patients, yet standardized risk scoring using unstructured electronic health records (EHRs) remains challenging. We retrospectively analyzed anonymized EHRs from a nationwide multicenter cohort across 30 hospitals. Using stratified random sampling, 50 cases from 10 hospitals were used for development and 200 cases from the remaining 20 hospitals for testing. Expert knowledge-augmented prompts for the Padua and Caprini scores were developed through a Delphi process involving 19 specialists and compared with basic and complex prompts. Performance was evaluated using six open-source large language models (LLMs), with prevalence-adjusted bias-adjusted kappa (PABAK) and F1 score as primary metrics. Expert knowledge-augmented prompts showed the highest performance in both datasets. In the test dataset, mean item-level PABAK and F1 scores were 0.97 and 0.96 for Padua, and 0.97 and 0.93 for Caprini; most items showed PABAK and F1 scores above 0.90. Risk stratification was better for Padua (PABAK 0.92; F1 score 0.96) than for Caprini (PABAK 0.73; F1 score 0.64), with processing times of 6.07 s and 12.82 s per case, respectively. These findings indicated that LLMs with expert knowledge-augmented prompting could efficiently support automated Padua and Caprini scoring and risk stratification in thrombosis prevention workflows.