HybridMolGen: A Unified Framework for Goal-Directed Molecular Generation via Multi-Objective Reinforcement Learning

Abstract Motivation De novo molecular design remains a fundamental challenge in drug discovery, requiring simultaneous optimization of multiple conflicting objectives such as drug-likeness, synthetic accessibility, and novelty while maintaining chemical validity. We present HybridMolGen, a novel unified framework that synergistically combines three complementary deep learning paradigms: (1) diffusion probabilistic models that generate high-quality, chemically valid molecular samples through gradual noise removal, (2) SE(3)-equivariant graph neural networks that enforce geometric and topological constraints ensuring structural validity and molecular diversity, and (3) property-conditioned transformers that enable fine-grained control over multiple objectives through multi-layer cross-attention modulation. Results These components operate within a multi-objective reinforcement learning paradigm that discovers optimal property trade-offs without manual weight tuning. Extensive benchmarking on MOSES, GuacaMol, and ZINC-250k datasets demonstrates state-of-the-art performance: 99.7% validity, 94.3% novelty, average QED score of 0.753, and 4.9% improvement in GuacaMol overall scores. Critically, HybridMolGen discovers 1.57× more molecules satisfying all target criteria simultaneously (91.3% vs 58.3% for CPRL) and generates 2.23× more Pareto-efficient solutions compared to traditional scalarization, demonstrating genuine architectural synergy beyond simple component aggregation. Comprehensive ablation studies confirm that the three-way integration outperforms even the best two-component combination by 6.5%, positioning HybridMolGen as a powerful tool for accelerating drug discovery pipelines. Availability and Implementation Implementation code are available as supplementary material. Supplementary Information Supplementary data are available at Bioinformatics online.