What question did this study set out to answer?

This research aims to improve cancer classification by integrating histopathological images and genomic data using a multimodal learning approach.

April 18, 2026Open Access

End-to-End Multimodal Multiple Instance Learning for Cancer Histopathology Classification with Dual-Attention Fusion

Key Points

This research aims to improve cancer classification by integrating histopathological images and genomic data using a multimodal learning approach.
Developed a lightweight end-to-end multimodal model incorporating whole slide images and gene expression profiles.
Used reduced MobileNetV4 for image feature extraction and Gene Set Enrichment Analysis to analyze gene information.
Employed self-attention and cross-attention mechanisms for feature aggregation across modalities.
Achieved ROC-AUC scores of 0.740 for low-grade glioma, 0.982 for non-small cell lung cancer, and 0.966 for breast cancer.
Notable improvement in PR-AUC for breast cancer classification was observed.
Demonstrated effectiveness of integrating morphological and molecular information for disease classification.

Abstract

In recent years, computational pathology has advanced toward integrating histopathological images and genomic data to improve diagnostic accuracy and biological interpretability. In this study, we propose a lightweight end-to-end multi-instance learning model that integrates whole slide images (WSIs) with gene expression profiles. The proposed method efficiently extracts image features using a reduced MobileNetV4 and treats gene information at the gene set level based on Gene Set Enrichment Analysis (GSEA) to capture functional relationships among genes. Image and gene features are aggregated within and across modalities through self-attention and cross-attention mechanisms. Our experiments on three cancer types, namely low-grade glioma (LGG), non-small cell lung cancer (NSCLC), and breast cancer (BRCA) showed that the proposed model outperformed image-only models and conventional single-vector gene input models. In particular, the ROC-AUC reached 0.740 for LGG, 0.982 for NSCLC, and 0.966 for BRCA, with a notable improvement in PR-AUC observed for BRCA classification. These results indicate that the integration of morphological and molecular information is effective for capturing disease characteristics. Furthermore, the proposed model maintains high classification performance while reducing computational resources, suggesting its potential applicability to large-scale pathological datasets and clinical applications.

Mark Helpful

Bookmark

Relay

View Full Paper