Validation of an AI-based automated Knee Inflammation MRI Scoring System for assessing bone marrow lesions

Objectives: Artificial intelligence (AI) offers potential to automatically evaluate the burden of active arthritis on magnetic resonance imaging (MRI) but must be reliable and practical to see routine use. We sought to validate the reliability and feasibility of iKIMRISS, an AI-automated iteration of the Knee Inflammation MRI Scoring System (KIMRISS) for bone marrow lesions (BMLs), using both quantitative and qualitative methods. Methods: Eleven readers participated in a 3-part reading exercise evaluating 40 2-time-point knee MRI cases using manual KIMRISS and different levels of AI automation in iKIMRISS. BML scoring reliability between methods was assessed using agreement metrics. A subset of experts also participated in postexercise questionnaires and semistructured interviews to assess the usability, feasibility, and implementation potential of iKIMRISS. Results: iKIMRISS' BML scores demonstrated moderate-to-strong agreement with human reader scoring, especially human readers who were experienced KIMRISS users (mean intraclass correlation coefficients of 0.76 for both baseline and time interval change scores). Thematic analysis of survey responses and semistructured interviews revealed an overall positive sentiment towards iKIMRISS as a research tool but indicated a need for further technical and logistical improvements to increase its value in clinical settings. Conclusions: iKIMRISS greatly improves the speed and accessibility of KIMRISS BML scoring while maintaining acceptable reliability for research and clinical trial applications. However, further refinement of the tool is recommended to improve its applicability in clinical settings.