A Transformer for Reaction-Aware Compound Explorations with GFlowNet in QSAR-Guided Molecular Design

In drug discovery tasks, achieving a balance between high biological activity toward therapeutic targets and synthetic chemical feasibility is critically important. While the recently proposed deep learning-based molecular generation models have enabled explorations of vast chemical spaces, most existing approaches do not consider synthetic routes for generated compounds. To address this issue, TRACE-GFN is proposed for molecular optimization; this method incorporates chemical reaction pathways into a quantitative structure–activity relationship (QSAR)-guided molecular design procedure. The method integrates a transformer model to explicitly learn chemical reactions with a generative flow network (GFlowNet) that efficiently samples diverse candidates. In benchmark experiments involving dopamine receptor D2 (DRD2), AKT serine/threonine kinase 1 (AKT1), and C-X-C motif chemokine receptor 4 (CXCR4), TRACE-GFN demonstrated the ability to identify compounds with high QSAR values while maintaining strong diversity, outperforming the existing molecular generation models. These results demonstrate that the proposed model can efficiently explore promising compounds while accounting for real-world chemical reactions. The source code is publicly available under an MIT license at https://github.com/sekijima-lab/TRACE-GFN.