MRI-endoscopy deep learning for predicting pathologic complete response and disease-free survival in locally advanced rectal cancer

To develop and validate a multimodal deep learning model that integrates post-neoadjuvant chemoradiotherapy (nCRT) rectal MRI and endoscopic images for prediction of pathologic complete response (pCR) and disease-free survival (DFS) in patients with locally advanced rectal cancer (LARC). This multicenter retrospective study included patients with LARC from two institutions who underwent nCRT followed by total mesorectal excision. Post-nCRT, preoperative multiparametric MRI (T2-weighted imaging, diffusion-weighted imaging, and contrast-enhanced T1-weighted imaging) and endoscopic images were analyzed using a multimodal deep learning framework incorporating graph neural networks and cross-attention to integrate complementary imaging features. Model performance was evaluated using AUC, accuracy, F1 score, calibration curves, Brier score, and decision curve analysis. DFS risk stratification was assessed using Kaplan-Meier analysis. A total of 288 patients were included. For pCR prediction, the multimodal model achieved AUCs of 0.876, 0.848, and 0.818 in the training, internal validation, and external validation cohorts, respectively, outperforming the MRI-only and endoscopy-only models. For DFS prediction, the corresponding AUCs were 0.853, 0.823, and 0.794. The model showed acceptable calibration and higher net clinical benefit across clinically relevant threshold probabilities. Kaplan-Meier analysis showed significant separation of DFS between the model-defined high-risk and low-risk groups (log-rank p < 0.01). A multimodal MRI-endoscopy deep learning model showed robust performance for predicting pCR and DFS after nCRT in patients with LARC. By integrating intratumoral and mucosal information, this approach may support preoperative response assessment and individualized risk stratification. Not applicable. Preoperative prediction of pathologic complete response and disease-free survival remains challenging in locally advanced rectal cancer. MRI–endoscopy multimodal deep learning consistently outperformed single-modality models in predicting pathologic complete response and disease-free survival risk. Multimodal deep learning may guide individualized treatment and organ-preserving strategies by improving preoperative response assessment and risk stratification in rectal cancer patients.