Multimodal Modeling of Cancer Subtypes Using Digital Pathology and DNA Sequencing

Background: Optimal oncology treatment depends upon detailed cancer subtyping and molecular characterization. However, due to small datasets, prior digital pathology work often grouped detailed histologic subtypes by anatomy. Furthermore, the role of digital pathology as a complement to DNA sequencing rather than a surrogate remains underexplored. This study develops machine learning for H&E whole-slide images (WSIs) to model detailed cancer subtypes, identify phenotypically-associated biomarkers, and evaluate the added information of digital pathology to clinical tumor sequencing. Methods: A pan-cancer cohort of 367,535 WSIs matching clinical sequencing data from 62,460 patients across 163 OncoTree subtypes was curated. The first model, Aeon, used a knowledge graph and transformer to classify subtypes from WSIs using self-supervised features. The second model, Paladin, tested the association of molecular features with tumor histopathologic features, conditioned on detailed tumor subtype. This conditioning critically allows the model to identify more detailed phenotypic subtypes than already clinically established. Two independent test sets were used. Results: Aeon achieved AUROC 0.992 across 163 detailed subtypes, outperforming a genomics-based classifier for 151 subtypes. Reclassified cancers of unknown primary (CUP) exhibited expected prognostic and genomic associations (corrected log-rank p<0.05, corrected Fisher’s p<0.05). For biomarker inference (single nucleotide variants, pathway-level alterations, and higher-order features), Paladin achieved AUROC ≥0.80 for 165 (5%) of 3,541, improving on benchmarks. Importantly, the model also identified phenotype associations for functional states of KEAP1 variants of unknown significance (VUS) in lung adenocarcinoma; VUS cases with H&E-inferred functional significance exhibited shorter overall survival (OS; log-rank p<0.01). For STK11, discordant H&E phenotyping and sequencing identified cases with occult phenocopying, supported by immunohistochemistry (H&E score higher for cases with STK11 loss on IHC; Mann-Whitney U (MWU) p<0.01) and STK11 RNA abundance (MWU p<0.01). This phenocopying had prognostic consequences: wildtype (WT) cases with STK11-mutant phenotype on H&E formed an intermediate prognostic group between STK11-mutant cases and WT cases without STK11-mutant phenotype (log-rank p=0.01). Conclusions: Digital pathology effectively classifies 163 granular tumor subtypes using H&E WSIs, advancing an order of magnitude from the prior state of the art, outperforming a genomics classifier, and clarifying CUP diagnoses. We further established the role of digital pathology in complementing DNA sequencing via VUS annotation and identification of phenocopying, with molecular validation of these phenotypes and prognostic implications. Citation Format: Kevin Michael Boehm, Madison Darmofal, Arfath Patha, Andrew Aukmerman, Raymond Lim, Evan Seffar, Tom Pollard, Natasha Rekhtman, Jason Chang, Jie-Fu Chen, Armaan Kohli, Darin Moore, JianJiong Gao, Georgios Asimomitis, Anika Begum, Hikmat Al-Ahmadie, Michael F. Berger, Nikolaus Schultz, Sohrab P. Shah, Francisco Sanchez-Vega. Multimodal modeling of detailed cancer subtypes and molecular features from >60,000 patients with co-registered H&E images and clinical tumor sequencing [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2026; Part 1 (Regular Abstracts); 2026 Apr 17-22; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2026;86(7 Suppl):Abstract nr 1292.