A multi-modal artificial intelligence model trained on H&E tissue images and clinical patient data for prediction of BRAF mutations in colorectal cancer.

e15161 Background: Analyzing genetic mutations is essential for personalized therapy decisions in oncology and beyond. In colorectal cancer (CRC), mutations in BRAF ( BRAF mut ) render patients eligible for targeted treatment with BRAF inhibitors and immunotherapy. BRAF mut is routinely analyzed via genomic sequencing, which, however, is costly and does not provide diagnostic results immediately. Reliable detection of BRAF mut status from histological images using artificial intelligence (AI) could provide a more accessible and faster diagnostic alternative. Previous approaches to image-based BRAF mut analysis, however, failed to reach a critical level of accuracy on independent test cohorts, with reported area under the receiver operating characteristic curve (AUROC) values typically below 0.8. Methods: We hypothesized that combining tissue image analysis with clinical patient data could improve diagnostic accuracy for BRAF mut . Following our hypothesis, we developed a multi-modal AI model on n = 426 CRC cases from the TCGA database using BRAF mut data, H&E-stained tissue images, and information on patient parameters that are readily available in clinical practice and that are known to be associated with BRAF mut frequency. The model was based on a vision transformer architecture incorporating pathology-specific information using a pretrained embedding model for encoding tissue image features. Model performance was validated on an independent hold-out TCGA cohort of n = 107 samples and an additional external cohort of n = 104 CRC samples from the CPTAC database. Results: Our multi-modal AI model reached an AUROC of 0.85 ± 0.02 on the TCGA hold-out test set. Notably, on the second, external CPTAC dataset the predictive accuracy level was similar (AUROC of 0.83 ± 0.02) indicating the robustness of our approach to generalize to different datasets. Additional analyses showed that the significant performance boost over previous approaches was gained by the multi-modal analysis setup, as predicting BRAF mut based on either tissue, or clinical data alone led to lower accuracy values. Explainability analysis of AI results confirmed that relevant biological features on the histological slides, mainly tumor areas, were considered by the model for decision making. Conclusions: The combination of H&E images with clinically readily available patient parameters enabled us to develop a model predicting BRAF mut in CRC that surpasses previous approaches in diagnostic accuracy on independent evaluation cohorts. This indicates that combining histological tissue information with clinical parameters can increase the ability of AI models to predict genetic variants from tissue and more generally highlights the potential of multi-modal deep learning approaches to enhance predictive AI models in digital pathology.