Interpretable Skin Cancer Diagnosis with Contrastive Language-Image Pre-training

Recent advances in machine learning and computer vision have significantly improved the performance of skin cancer diagnostic models. However, their lack of interpretability poses a challenge for clinical adoption, as physicians may find it difficult to trust a diagnosis made by a “black box” system. We propose a novel methodology for skin cancer diagnosis using Contrastive Language-Image Pretraining (CLIP), allowing physicians to provide a set of features in natural language and then determine the weight our model gave each feature in its diagnosis. This approach aims to bridge the communication gap between physicians and machine learning models. We show that the CLIP model is able to diagnose skin cancer in a zero-shot setting and provide insight into how each provided feature contributes to its diagnosis.