Answers are grounded in this paper and cite their sources.
Background: Computer-Aided Diagnosis systems can support physicians by reducing workload, medical errors, and time required for patient care. However, many existing systems primarily rely on medical images while overlooking valuable information available in associated medical reports. This study proposes a multimodal deep learning approach for chest disease classification by integrating chest X-ray images and radiology reports from the MIMIC-CXR dataset. (2) Methods: Different Deep Learning models are evaluated for each modality, followed by a comprehensive comparison of intermediate and late fusion strategies to investigate the effectiveness of combining visual and textual information. The proposed framework also addresses practical challenges associated with multimodal medical data, including generating disease labels for unannotated data, handling class imbalance, aggregating multiple images at the study level, selecting an appropriate radiology report section for textual input, and managing missing textual information. Furthermore, unlike existing fusion strategies that rely on traditional fusion strategies or combine features into a single representation, the proposed label-aware intermediate fusion employs disease-specific queries to guide multimodal feature fusion, generating disease-specific representations for predicting the 14 disease labels. (3) Results: The best-performing approach combines visual features extracted from chest X-ray images using the Swin Transformer with textual features extracted from the Findings section of radiology reports using Bio-Clinical BERT through a label-aware intermediate fusion mechanism. This approach achieved a macro-AUC of 90.7% and a micro-AUC of 94.4%, demonstrating its effectiveness in chest disease classification. (4) Conclusions: The results demonstrate the value of combining complementary visual and textual information for chest disease classification and highlight the importance of selecting an appropriate fusion strategy for multimodal medical data. Beyond its diagnostic contribution, the study illustrates how a biological principle—hierarchical integration of complementary sensory channels, with compensation under modality loss—can be operationalized and evaluated within a clinical learning architecture. Ultimately, these efforts aim to improve, develop, and deliver healthcare services in an integrated manner.
No takes yet. Share an insight, caveat, or question.
Almatrafi et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: