Visual-Language Artificial Intelligence System for Knee Radiograph Diagnosis and Interpretation: A Collaborative System with Humans

Background Large language models (LLMs) have shown promising abilities in text-based clinical tasks, but they do not inherently interpret medical images such as knee radiographs. Purpose To develop a human-artificial intelligence (AI) interactive diagnostic approach, named radiology generative pre-trained transformer (RadGPT), aimed at assisting and synergizing with human users for the interpretation of knee radiological images. Materials and Methods A total of 22,512 knee X-ray images and reports were retrieved from Massachusetts General Hospital; 80% of these were used for model training and 10% were used for model testing and validation, respectively. Fifteen diagnostic imaging features (e.g. osteoarthritis, effusion, joint space narrowing, osteophyte) were selected to label images based on their high frequency and clinical relevance in the retrieved official reports. AUC scores were calculated for each feature to assess the diagnostic performance. To evaluate the quality of the generated medical text, historical clinical reports were used as the reference text. Several metrics for text generation tasks are applied, including BLEU (BiLingual Evaluation Understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), METEOR (Metric for Evaluation of Translation with Explicit Ordering), and SPICE (Semantic Propositional Image Caption Evaluation). Results RadGPT, in collaboration with human users, achieved AUC scores ranging from 0.76 for osteonecrosis to 0.91 for arthroplasty across 15 diagnostic categories for knee conditions. Compared with the baseline LLM method, RadGPT achieved higher scores, specifically 0.18 in BLEU score, 0.30 in ROUGE-L, 0.10 in METEOR, and 0.15 in SPICE, which is significantly higher than the baseline LLM method, demonstrating good linguistic overlap and clinical consistency with the reference reports. Conclusion RadGPT has achieved advanced results in knee X-ray image feature recognition, illustrating the potential of LLMs in medical image interpretation. The study establishes a training protocol for developing AI-assisted tools specifically focusing on the diagnosis and interpretation of knee radiological images.