Large language model-based description is all you need: Zero-shot medical concept embeddings using Transformer encoders

Abstract Concept embeddings enable knowledge to be integrated into AI models. This work investigates whether Transformer-based encoders, without training on the knowledge graph, can generate competitive medical concept embeddings. We generate contextual descriptions for each medical concept from SNOMED CT using a generative language model, ensuring the concept label is included within an informative text. These are then encoded using Transformer-based encoders to produce embeddings. To evaluate concept embeddings quality, we propose two benchmarks: Hierarchical Similarity, which assesses whether embeddings reflect SNOMED CT’s ontology, and Semantic Composition, which tests if composed embeddings recover expected concepts. Despite no access to the SNOMED CT graph, the embeddings produced by the encoders in a zero-shot manner outperform graph-based methods. These findings reveal that modern encoders can encode complex domain knowledge, offering a powerful alternative to graph-trained embeddings. The best-performing embeddings and benchmarks are released.