MT-ConBiFormer-GPT: multi-target molecular generation for low-data drug discovery via a contrastive BiFormer-GPT architecture and curriculum learning with cross-domain generalization

Abstract Multi-target compounds, or polypharmacological agents, hold significant potential for complex diseases like cancer, where single-target therapies are often insufficient. A lack of high-quality bioactivity data limits progress in this field, especially for compounds interacting with multiple proteins simultaneously. This study introduces MT-ConBiFormer-GPT, a deep generative model designed explicitly for low-data, multi-target molecular generation, focusing on the critical PI3K–AKT–mTOR cancer signaling pathway. The framework integrates a variational autoencoder with a BiFormer encoder to capture long-range dependencies in SMILES strings, reducing the quadratic computational complexity associated with standard transformers and mitigating semantic discontinuities. It employs a SMILES-GPT decoder for progressive molecule generation and follows a three-phase training pipeline: unsupervised pre-training, supervised contrastive learning, and curriculum-based fine-tuning. The framework’s efficacy was evaluated through a rigorous, multi-stage assessment. First, the framework was evaluated through benchmarking against state-of-the-art models, with a specialized head-to-head variant, MT-ConBiFormer-GPTH2H, demonstrating superior performance, thereby validating its generalizability from oncology to neuropsychiatry. An internal ablation study further revealed that the full MT-ConBiFormer-GPT significantly outperformed its baseline, MT-BiFormer-GPT, in both dual- and triplet-target generation tasks, highlighting the advantages of the contrastive learning stage. Additionally, the foundational Base-BiFormer-GPT architecture, a model lacking both the contrastive and curriculum learning stages, highlighted its intrinsic robustness by achieving competitive outcomes in a distinct omics-driven design task. Docking simulations and mechanistic analyses show that the generated molecules, including high-fidelity and scaffold-hopping candidates, display more favorable binding modes than reference inhibitors. This study presents a flexible and computationally efficient framework for multi-target drug discovery in data-limited settings.