Knowledge-driven multimodal medical image classification

Medical image classification is significantly hindered by the scarcity of data for minority classes, which leads to a major classification bias in models. This issue results in poor and unreliable performance on underrepresented, yet often critical, stages of disease. To address this issue, we propose a knowledge-driven multimodal classification model that integrates domain-specific images and texts. Specifically, we design a visual knowledge cache to store and retrieve representative lesion features. Meanwhile, a textual knowledge cache is proposed to dynamically select semantically relevant textual descriptions closely aligned with visual features. Furthermore, we introduce a multimodal knowledge reasoner that combines image-to-image retrieval and image-to-text alignment, leading to improved fine-grained disease classification. We evaluate our method on three public datasets APTOS2019 , BUSI and COVID × CT-3 A . Extensive experiments demonstrate that our model achieves superior grading accuracy, with an average a 2.90% improvement in accuracy, highlighting its effectiveness in medical image classification.