Multimodal Prediction of Renal Tumor Malignancy from Radiology Reports and Structured Electronic Health Records: Retrospective Cohort Study (Preprint)

BACKGROUND Accurate preoperative prediction of renal tumor malignancy is essential but remains challenging. While radiology and structured electronic health record (EHR) data are widely used for tumor evaluation, radiology reports—though rich in diagnostic information, have been relatively underutilized due to extraction challenges. OBJECTIVE To enhance malignancy prediction of renal tumors by developing a multimodal pipeline that integrates structured EHR data with information extracted from radiology report text using advanced large language models. METHODS We developed a predictive framework that integrates structured EHR variables with features derived from radiology reports. Large language models (LLMs) were used to extract abnormality characteristics, and a pretrained biomedical transformers (i.e., RadBERT) was applied to generate contextual embeddings from unstructured radiology text. These textual features were fused with structured data using early, middle and late fusion strategies. Model performance was evaluated using standard classification metrics, including accuracy, precision, recall, specificity, area under the ROC curve (AUC) and F1-score. RESULTS Incorporating RadBERT-derived textual features improved prediction performance, while LLM-extracted abnormality characteristics contributed modest gains. Among fusion strategies, early fusion achieved the highest AUC of 0.818 (± 0.010), and late fusion provided the best F1-score of 0.779 (±0.022), both outperforming unimodal baselines. CONCLUSIONS Our findings demonstrate the value of leveraging unstructured radiology reports through advanced natural language processing (NLP) techniques for malignancy predication. This multimodal fusion approach enhances preoperative renal tumor assessment and has the potential to reduce unnecessary surgeries and improve patient care.