Motivation: While currentartificial intelligence (AI) applications in this domain primarily focus on single-modality imaging,such as mammography or MRI, these approaches are limited in capturing the full complexity of breast cancer's heterogeneity Goal(s): Developed a deep learning-based model for predicting molecular subtypes of breast cancer through diagnostic mammography (MG) and MRI image. Approach: We implemented a multimodal deep learning architecture,incorporating a cross-attention mechanism for MG and self-attentionfor MRI. Results: This deep learning model demonstrates superior predictive accuracy by lever-aging both 2D MG and 3D MRI data, making it a valuable non-invasive tool for molecular subtype identification. Impact: By uniquely combining 2D mammography and 3D MRI data, the multimodal deep learning model captures complementary tumor characteristics, supporting more accurate and nuanced classification across multiple subtypes, potentially aiding in treatment planning and improving patient outcomes.
He et al. (Tue,) studied this question.