Abstract In recent years Vision language models (VLM) have significantly gained popularity to process and analyze medical imaging data. The features extracted by vision encoders from the VLMs can be used in different activities like embedding with relevant text prompts or forming unsupervised clusters to find hidden biomarkers. The major challenge in digital pathology is that a huge portion of available datasets are unlabeled or partially labelled which makes the segmentation or classification tasks difficult. Unsupervised learning is used to find hidden patterns from the unlabeled Whole Slide Images (WSIs) by extracting image embeddings and by finding significant features as potential biomarkers. In this study we introduce "Unsupervised VLM-Driven Biomarker Identification and Meta-Integration (UVLMBI)," a novel framework that is designed to leverage VLM capabilities in unsupervised biomarker discovery and integration against relevant clinical metadata. Our method is used to extract rich feature embeddings from whole slide images of LungMAP dataset, devoid of explicit labels, with an available sparse metadata. We applied a suite of state-of-the-art VLMs, with most of them pre-trained on pathology-related image datasets (UNI, Prov-Gigapath, Conch, and Plip), while some of the VLMs were used as generalized foundation model applicable to medical imaging data (such as Llava and Llava-next). Our approach begins with extraction of image embeddings from the image encoders of each VLMs. Subsequently multiple unsupervised clustering algorithms, such as K-Means, Agglomerative, Gaussian Mixture, and Spectral are applied on the embeddings. Based on the clustering performances, we identify the optimal clustering – VLM combinations for the later experiments. One key novelty of the UVLMBI is the integration of clinical metadata to enhance the biomarker discovery. We develop predictive models to identify relevant biomarkers by linking image-derived clusters to clinical metadata based on patient demographics, clinical history, and molecular data. Beyond the mere identification of putative biomarkers, the integration delivers information about their clinical relevance in the light of personalized medicine strategies. Our experimental results show the effectiveness of UVLMBI framework in finding clinically significant biomarkers and provide comprehensive evaluations for collections of VLMs in unsupervised medical image analysis. This framework opens new dimensions of advanced data-driven techniques in digital pathology and opens new ways of research and clinical applications. This study provides an exploratory investigation of using VLMs for unsupervised biomarker discovery, acknowledging the challenges in clustering performance with complex medical image datasets. Our experimental results on a set of WSIs from human developing lung tissues in combination with clinical metadata shows promising results and can lead to unsupervised biomarker discovery in computational pathology.
Debsarkar et al. (Sun,) studied this question.