Deep learning (DL) is revolutionizing high-dimensional analysis for clinical applications, with multimodal approaches showing promise in integrating diverse datasets and improving predictive accuracy. However, its use in prognostic assessment, particularly for non-small-cell lung cancer (NSCLC), remains limited. Whole-slide imaging (WSI), next-generation sequencing (NGS), and clinical features from 2,366 NSCLC patients sourced from various databases were analyzed. A Sequential-Adapted Attention (SeAttn) model was developed to process WSI, while multiple DL models analyzed genetic and clinical data. Multimodal features were combined using a COX-based DL approach to predict long-term prognosis. Validation (N = 100) and test (N = 101) cohorts were used to confirm model performance. The SeAttn model achieved an area under the curve (AUC) of 0.98 for NSCLC histologic subtype prediction in the test cohort (P < 0.0001). Individually, WSI, clinical, and genetic features predicted prognosis with concordance indices (C-indices) of 0.55–0.67 using the test cohort. Combined multimodal features improved the test cohort C-index to 0.71 (P = 0.002). The model accurately predicted prognosis up to 5 years post-diagnosis (P < 0.05 for all time-dependent AUCs) and stratified patients by overall survival (P = 0.012). SeAttn demonstrated attention shifts when comparing subtyping and prognosis models and revealed immuno-hot and immuno-cold stroma enrichment underlying survival differences. Novel prognostic markers, including TPTE mutation, microRNA cluster amplification, and RIPK4-IL10RB co-deletion, were identified and validated. This multimodal DL framework demonstrates robust prognostic capabilities for NSCLC, integrating clinical, genetic, and imaging data to yield accurate survival estimates and identify biomarkers. These findings highlight the scalability of multimodal DL for broader cancer applications.
Wang et al. (Tue,) studied this question.