PURPOSE Multimodal machine learning offers a holistic view of a patient's status, integrating structured and unstructured data from electronic health records (EHR). We propose a framework to predict metastasis risk 1 month before diagnosis, using 6 months of clinical history from EHR data. METHODS Data from four cancer cohorts collected at Karolinska University Hospital (Stockholm, Sweden) were analyzed: breast (n = 743), colon (n = 387), lung (n = 870), and prostate (n = 1,890). The data set included demographics, comorbidities, laboratory results, medications, and clinical text. We compared traditional and deep learning (DL) classifiers across single modalities and multimodal combinations, using various fusion strategies and a transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) 2a design, with an 80-20 development-validation split to ensure a rigorous, repeatable evaluation. Performance was evaluated using AUROC, area under the precision-recall curve, F1 score, sensitivity, and specificity. We then employed a multimodal adaptation of Shapley additive explanations (SHAP) to analyze the classifiers' reasoning. RESULTS Intermediate fusion achieved the highest F1 scores on breast (0.845), colon (0.786), and prostate cancer (0.845), demonstrating strong predictive performance. For lung cancer, the intermediate fusion achieved an F1 score of 0.819, while the text-only model achieved the highest, with an F1 score of 0.829. DL classifiers consistently outperformed traditional models. Colon cancer, the smallest cohort, had the lowest performance, highlighting the importance of sufficient training data. SHAP analysis showed that the relative importance of modalities varied across cancer types. CONCLUSION Fusion strategies offer distinct strengths and weaknesses. Intermediate fusion consistently delivered the best results, but strategy choices should align with data characteristics and organizational needs.
Rugolon et al. (Wed,) studied this question.