A Multimodal Deep Learning Model Integrating CNN and Transformer for Predicting Chemotherapy-Induced Cardiotoxicity

Chemotherapy-induced cardiotoxicity presents a major risk to cancer patients, often leading to severe cardiac complications such as heart failure, myocardial infarction, and arrhythmias. Early detection is crucial for preventing long-term damage and improving patient outcomes, yet existing diagnostic methods struggle to effectively capture the complexity of multimodal medical data and often lack interpretability. In this study, we propose an innovative approach that integrates multimodal deep learning with Explainable AI (XAI) techniques to enhance early cardiotoxicity detection. Our model combines clinical data (e.g., age and cardiovascular metrics) with Tissue Doppler Imaging (TDI), a functional imaging technique that captures myocardial velocity during the cardiac cycle. To overcome data limitations, we employed Conditional Generative Adversarial Networks (cGANs) and Conditional Tabular Generative Adversarial Networks (CTGANs) to augment the dataset, improving its diversity and balance for better model training. We developed three architectures that integrate Convolutional Neural Networks (CNNs) for feature extraction from TDI images with Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Transformer models to capture temporal dependencies and enhance prediction accuracy. Additionally, we incorporated SHapley Additive Explanations (SHAP) to interpret the contribution of input features, increasing model transparency and clinical applicability. Our Transformer-based model achieved the highest accuracy of 96%, outperforming the GRU (94%) and LSTM (89%) models, significantly surpassing traditional approaches. These findings highlight the potential of transformer-based architectures in multimodal deep learning for precise cardiotoxicity prediction, supporting early intervention and personalized treatment strategies while improving interpretability through XAI techniques such as SHAP.