Answers are grounded in this paper and cite their sources.
Chest X-ray plays a crucial role in the screening and diagnosis of chest diseases. Due to the complexity of pathological manifestations and limitations of radiologists' experience, the accuracy and efficiency of diagnosing chest diseases need to be further improved. In recent years, deep learning has made significant progress in chest X-ray image analysis, while existing methods mainly rely on uni-modal visual information, overlooking the prior knowledge related to disease category descriptions embedded in medical text data, making it challenging to fully understand the deep semantics of chest X-ray images. To address these challenges, inspired by the Instruction-ViT model, we adopt instruction tuning techniques to integrate medical textual information into the fine-tuning process of the visual model. Furthermore, a contrastive learning loss is employed to align textual and visual features, thereby enhancing the model's capacity to understand and differentiate complex pathological patterns. Experimental results demonstrate that the model integrating medical text information outperforms uni-modal models in various evaluation metrics, confirming that with instruction tuning, our model can effectively utilize medical text as prior knowledge to improve the performance of visual models in chest disease diagnosis. Furthermore, we conduct an interpretability analysis of the model's decision-making process, revealing that the regions attended to by the model highly correspond to the radiographic manifestations of different diseases, demonstrating the model's interpretability to a certain degree. • Multi-modal Instruction Tuning: Fuses X-ray and clinical text with contrastive learning for context-aware diagnosis. • Superior Diagnostic Performance: Outperforms uni-modal models on MIMIC-CXR and SIIM-ACR, boosting thoracic disease accuracy. • Enhanced Interpretability: Attention maps align with radiographic findings, validating decisions and clinical relevance.
No takes yet. Share an insight, caveat, or question.
Yao et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: