Embedding of differential expression data using the contrastive language-image pre-training objective

In a given cellular context, such as a disease state, transcription may be altered. Differential expression analysis aims to quantify differences in transcript abundance across genes between conditions, but the current methods for interpretation of its results struggle to generate meaningful scientific insight and barely utilize the large amounts of gene expression data generated by the scientific community. In an effort to improve the interpretability, I utilized the Contrastive Language-Image Pre-training (CLIP) objective in order to create a joint embedding space for numerical differential expression data and their textual annotations. As training data, I used Gemma, a database containing tens of thousands of publicly available expression experiments that have been re-analyzed for differential expression. I trained embedding models from scratch but also fine-tuned the single-cell foundation models Geneformer and scGPT, which surprisingly resulted in an increase in performance. I thereby demonstrated that single-cell foundation models are able to extrapolate information from their pre-training to bulk differential expression data despite the fundamental difference in data modality. My embeddings allow identification of biological conditions that have characteristic transcriptomic signatures.