An Integrated Multi-modal Learning Method for Early-stage Knee Osteoarthritis Disease Classification

Abstract Background : Osteoarthritis (OA) is a progressive and chronic disease. Identifying the early stages of OA disease is important for the treatment and care of patients. However, most state-of-the-art methods only use single-modal data to predict disease status, so that these methods usually ignore complementary information in multi-modal data. Methods : In this study, we develop an integrated multi-modal learning method (MMLM) that uses an interpretable strategy to select and fuse clinical, imaging, and demographic features to classify the grade of early-stage knee OA disease. MMLM applies XGboost and ResNet50 to extract two heterogeneous features from the clinical data and imaging data, respectively. And then we integrate these extracted features with demographic data. To avoid the negative effects of redundant features in a direct integration of multiple features, we propose a L1-norm-based optimization method (MMLM) to regularize the inter-correlations among the multiple features. Results : MMLM was assessed using the Osteoarthritis Initiative (OAI) data set with machine learning classifiers. Extensive experiments demonstrate that MMLM improves the performance of the classifiers. Based on MMLM, the accuracy of SVM and DT are 83.45% and 81.27%, respectively, which are much higher than other feature fusion method. In addition, the visual analysis of the important features in the multi-modal data verified the relations among the different modalities. Conclusion : MMLM uses the internal correlations among different modalities to enhance the efficiency of feature extraction and fusion, which improves grade classification of early-stage knee OA.