MSAT-KOA-Net: a multi-scale attention transformer network with retrieval-augmented knowledge (RAG) for interpretable knee osteoarthritis grading

Knee Osteoarthritis (KOA) has become the leading degenerative joint problem affecting humans across the world, causing disability among the elderly population. The radiographic evaluation of KOA is often performed using the Kellgren–Lawrence grading system; however, grading is subjective and depends on expertise. Recently, deep learning (DL) models have shown promising results for KOA diagnosis; however, these approaches have limitations in representing multi-scale anatomical features, focusing on joint regions of interest, modelling global relationships between features, and interpretability of results. In this paper, a novel Multi-Scale Attention Transformer Network (MSAT-KOA-Net) architecture has been proposed for accurate grading of knee osteoarthritis from radiographic images while maintaining interpretability of results. The proposed architecture combines an EfficientNet-B4 backbone network with a multi-scale feature fusion layer, a Convolutional Block Attention Module (CBAM), and a joint space attention mechanism to achieve discriminative feature representation of knee joint structures. Furthermore, a transformer encoder is employed to encode global contextual relationships between the extracted features. Additionally, an ordinal focal loss function is used to effectively learn the ordered relationships of KL grades and address the class imbalance issue. In addition to classification, a Retrieval-Augmented Generation (RAG) module is employed to improve the model’s interpretability by retrieving relevant medical knowledge from scientific literature via FAISS-based semantic search and generating contextual explanations with a large language model. The proposed MSAT-KOA-Net framework performance is evaluated with the knee X-ray dataset, and the results show that the proposed framework outperforms the baseline models with a validation accuracy of 80.09% and a Quadratic Weighted Kappa score of 0.8523, thus validating the effectiveness of the proposed framework for improving the classification of KOA severity and providing relevant explanations for the decision-making process, which is very helpful for the diagnosis of osteoarthritis.