Multimodal deep learning for prediction of postoperative recurrence in clear cell renal cell carcinoma: a clinical-radiologic-pathologic approach

Purpose: This study intends to develop and validate a multiclass system that efficiently integrates complementary information from patients' multimodal data (clinical, imaging, and pathological) to identify individuals at a high risk of 5-year postoperative recurrence among patients with clear cell renal cell carcinoma (ccRCC). Methods: This retrospective multicenter study enrolled 270 clear cell renal cell carcinoma (ccRCC) patients, allocating 215 (79.6%) for model development/validation and 55 (20.4%) to an independent test cohort. Patients were categorized into recurrence and non-recurrence groups based on whether tumor recurrence occurred within 5 years after surgery. The final included case in this cohort underwent surgery in July 2018. Single-modality models were created using radiomics, deep learning (ResNet34), and deep features-based radiomics. Subsequently, multi-modality models were constructed by combining the predicted probabilities from the single-modality models and transferring them to another classifier. All models was assessed using the area under the receiver operating characteristic curve (AUROC). Results: A total of 25 classification models were constructed. Notably, single-modality fusion models generally outperform their radiomics and deep learning-based radiomics (DL) counterparts. Among five single-modality fusion models, the Fused_Non-enhanced model demonstrates best predictive performance, achieving an AUROC value of 0.837 (95% confidence interval [CI]: 0.729-0.946). Similarly, multi-modality radiomics or DL models exhibit superior performance compared to single-modality counterparts. The multi-modality radiomics-DL model demonstrates the highest prediction performance, achieving an AUROC value of 0.967 (95% CI: 0.929-1.0) in the independent testing dataset. Conclusion: The multi-modality radiomics-DL model demonstrates high accuracy in predicting the 5-year postoperative recurrence risk of clear cell renal cell carcinoma (ccRCC).