Automatic Radiology Report Generation (RRG) reduces medical workload by automatically generating diagnostic reports from X-ray images. In recent studies, an improved performance has been observed by first classifying the image and then incorporating clinical outcomes into the RRG pipeline, with cross-modal interactions between image and text. However, these studies have limitations, including : (1) inconsistent cross-modal learning due to under-utilization of textual information, and (2) reliance on noisy classification labels, resulting in inaccurate attention and invariant report generation. To address these issues, a Medical Context-aware RRG Framework (MedCARF) is proposed, which introduces a Disease-Aware Visual Textual Alignment (D-ViTAlign) module to capture organ-abnormality relationships from scene graphs and visual features. In addition, label refinement is achieved using an unsupervised Augmentation Distillation with Contrastive Refinement (AuDiCoR) loss, which iteratively refines the visual features via augmentation alignment and loss modelling. Furthermore, the knee OAI dataset is used to benchmark the RRG task, and the contributions of MedCARF are tested on it. Later, Knee RRG performance is also evaluated using multimodal information, including biomechanical data. Experimental results showed that a minimum 3.2% improvement in METEOR and a 4.22% AUC gain over classification SOTA are achieved for chest X-ray, while a 9.69% average improvement across all NLG metrics is observed for knee RRG with biomechanical data integration. Overall, contextually accurate reports are generated, and consistent classification performance is achieved across minority classes, thereby demonstrating the robustness and clinical applicability of the proposed framework. The Code is available at: https://github.com/adaydar/MedCARF.
Daydar et al. (Thu,) studied this question.