MEMOL: Mixture of experts for multimodal learning through multi-head attention to predict drug toxicity

Accurate toxicity prediction is essential in drug development, but several challenges remain due to the high complexity of molecular mechanisms. In this regard, toxicity prediction methods using deep learning have emerged as promising solutions, with multimodal approaches by integrating diverse molecular modalities. However, an effective fusion of multiple modalities to achieve precise toxicity predictions still requires further exploration. In this study, we propose MEMOL, mixture of experts for multimodal learning through multi-head attention. This novel framework integrates modalities of molecular images, graphs, and fingerprints, and applies a sparse mixture of experts directly into the attention mechanisms. In addition, MEMOL enhances feature extraction and modality fusion through self- and cross-attention mechanisms, leading to improved predictive performance and efficiency. A comparative experiment on several toxicity benchmark datasets demonstrated that MEMOL outperformed state-of-the-art methods, achieving up to 8.33% higher AUROC and 9.11% higher AUPRC than the second-best model. Furthermore, the multimodal learning analysis showed that the employment of all three modalities consistently resulted in considerable performance improvements, and the ablation study revealed that a sparse mixture of experts enabled effective multimodal integration for drug toxicity prediction. MEMOL demonstrates superior performance in toxicity prediction, achieving consistent improvements across diverse datasets and offering a robust framework for integrating molecular representations. • MEMOL effectively integrates three molecular modalities—images, graphs, and fingerprints—to provide a more comprehensive and robust representation of molecules. • By applying a Mixture of Experts to both self- and cross-attention, MEMOL can precisely learn both intra- and inter-modality relationships. • Since MEMOL adopts a sparse Mixture of Experts instead of a dense Mixture of Experts, it can predict the toxicity of molecules more accurately and efficiently. • MEMOL, as a state-of-the-art model, achieved up to 8.33% higher AUROC and 9.11% higher AUPRC compared to other toxicity prediction models.