A Multimodal Transfer Learning Approach for Medical Image Classification

In machine learning for healthcare, medical image data often suffers from data scarcity and expensive annotation processes. In this study, to develop a robust medical image classification model, we propose a novel approach that leverages the multimodal pre-trained PubMedCLIP model as a backbone. Through experiments, we demonstrate that this straightforward approach offers substantial benefits in image classification task, reducing annotation costs while maintaining performance. By leveraging multimodal information and the power of pre-trained models, our method provides a promising solution to address the challenges posed by limited medical image data, enabling more efficient and accurate image classification.