Deep learning analysis of MRI to assess rectal cancer treatment

Introduction Traditional neoadjuvant therapy for locally advanced rectal cancer (LARC) results in pathologic complete response (pCR) in approximately 15% of patients, supporting non-operative strategies for those with clinical complete response (cCR). The subjectivity and variability in MRI-based cCR assessments highlight the need for objective, quantitative tools. Objective To develop deep learning models for automated rectal tumor segmentation on pre- and post-treatment MRIs, and to identify radiomic features differentiating cCR from non-cCR patients. Materials and methods We retrospectively analyzed pre- and post-treatment MRIs from 37 LARC patients enrolled in a Phase 2 TNT trial (NCT04380337). Rectal tumors were segmented on T2-weighted images by two data scientists, refined by a radiologist (reference standard), and independently segmented by a fellow. For pre-treatment segmentation, Model 1 (baseline; n = 37 ) was trained on reference cases, then used to generate pseudo-labels for 81 additional cases. Model 2 (semi-supervised; n = 118 ) was trained on the combined dataset. Model 3 (baseline; n = 37 ) was trained on post-treatment cases. Radiomic features were extracted from post-treatment ADC maps, filtered by reproducibility (ICC ≥ 0.8 ) and redundancy (Spearman ρ ≤ 0.95 ), then analyzed using unsupervised hierarchical clustering. Results For pre-treatment segmentation, radiologist-fellow inter-rater agreement was DSC = 0.748 ± 0.092 . Model 1 achieved mean DSC = 0.682 ± 0.254 versus the radiologist, significantly lower than inter-rater agreement. Model 2 improved performance to mean DSC = 0.769 ± 0.214 (mean gain = 0.087 ; 12.8 % relative improvement; p < 0.001 ), slightly outperforming inter-rater agreement. For post-treatment segmentation, inter-rater agreement declined to mean DSC = 0.362 ± 0.256 , while Model 3 achieved mean DSC = 0.175 ± 0.231 versus the radiologist, reflecting challenges from treatment-induced tissue changes affecting both automated models and human raters. Radiomic clustering revealed two distinct patient groups aligned with cCR and non-cCR status. Conclusion This study demonstrates the feasibility of deep learning-based automated segmentation and radiomic profiling for differentiating treatment response in rectal cancer. Semi-supervised learning with pseudo-labeled data significantly improved segmentation performance, offering a practical approach to overcome limited annotations. Radiomic features warrant validation in larger multi-center studies for clinical translation.