CSIP: Contrastive learning with Graph Neural Network enables chemical structure to image mapping

The development of advanced AI has been transforming the biomedical field. The emergence of multi-modal biomedical data such as imaging, sequencing, and other omics data further enabled the training of AI models for complex analytical tasks. However, such data analysis remains challenging due to the difficulty of integrating information from different modalities. In this paper, we aim to address this challenge by proposing a methodology to integrate image and molecular structure data. We present the Contrastive Structure-to-Image Pretraining (CSIP) framework, which leverages a self-supervised Graph Neural Network (GNN) to encode molecules and images into a joint feature embedding space. This direct mapping between the two modalities enables a wide variety of applications, including image profiling and clustering of molecules based on their effects on cell morphology. Image profiles generated by CSIP archived an average AUC of 0.708 on various biological activity prediction tasks, rivalling the state-of-the-arts and outperforming some fully supervised methodologies. Further, CSIP improved the accuracy of image-molecule matching by 29-folds from the random baseline after being trained on a small dataset, which demonstrated data efficiency. The code to reproduce our results can be found at https://github.com/LeoL18/CSIP.