MorganBERT: Chemical substructure sequences meet language models for property prediction

Language models applied to text representations of molecules have emerged as powerful tools for modeling chemical structures and their connections with biological activities. However, the utilization of prevailing line notations, coupled with limited molecular property labels, hinder the exploration of the vast chemical space. Here we present MorganBERT, a suite of chemical language models pretrained on corpora of string representations of 52 million compounds. Using a molecular fingerprinting algorithm to generate chemical substructure identifiers, the corpora for pretraining consist of identifier sequences obtained by traversing chemicals. We demonstrate the effectiveness of MorganBERT models by comparing with diverse categories of methods across multiple tasks relevant to drug development and toxicology. Further analysis of the vector representations of chemicals reveals that MorganBERT can extract an approximate isometry between chemical space and activity space. Leveraging this unique advantage, we showcase the explainability of MorganBERT predictions for individual chemicals through counterfactuals. The pretrained models are available at https://huggingface.co/Keylab.