Answers are grounded in this paper and cite their sources.
Visual representation learning plays a key role in solving medical computer vision tasks. Recent advances in the literature often rely on vision-language models aiming to learn the representation of medical images from the supervision of paired captions in a label-free manner. The training of such models is however very data/time intensive and the alignment strategies involved in the contrastive loss functions may not capture the full richness of information carried by inter-data relationships. We assume here that considering expert knowledge from the medical domain can provide solutions to these problems during model optimization. To this end, we propose a novel knowledge-augmented vision-language contrastive representation learning framework consisting of the following steps: (1) Modeling the hierarchical relationships between various medical concepts using expert knowledge and medical images in a dataset through a knowledge graph, followed by translating each node into a knowledge embedding; And (2) integrating knowledge embeddings into a vision-language contrastive learning framework, either by introducing an additional alignment loss between visual and knowledge embeddings or by relaxing binary constraints of vision-language alignment using knowledge embeddings. Our results demonstrate that the proposed solution achieves competitive performances against state-of-the-art approaches for downstream tasks while requiring significantly less training data. Our code is available at https://github.com/Wxy-24/KL-CVR . • Rich semantics can be developed in representation learning by leveraging. • The inter-data relationships can be modelled using external knowledge. • Additional cross-modal alignment leads to better visual understanding. • Fine-grained inter-data similarity can serve as soft targets for cross-modal alignment.
No takes yet. Share an insight, caveat, or question.
Wei et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: