Evaluation of large language models’ responses to patient questions about avascular necrosis of the femoral head

Artificial intelligence chatbots are rapidly gaining popularity for acquiring health information, but the quality of responses seems to differ between platforms. This study investigated the responses of ChatGPT, Gemini, Copilot, Claude, and DeepSeek to 15 frequently asked questions regarding femoral head avascular necrosis. Each platform answered each question once, for a total of 75 responses. Four orthopedic surgeons assessed the response using the JAMA benchmark criteria, DISCERN, an adapted five-point scale of clinical accuracy, and a binary safety score. Readability was measured by Flesch reading ease score (FRES) and Flesch-Kincaid grade level (FKGL). Models were compared by using Friedman tests and Bonferroni-corrected Wilcoxon signed-rank tests. Claude ranked the highest by JAMA and DISCERN scores and that DeepSeek outperformed ChatGPT, Gemini, and Copilot in terms of DISCERN. Clinical accuracy differed significantly among the models ( p < 0.001). Claude and DeepSeek were rated higher than ChatGPT, Gemini, and Copilot, with no significant difference between Claude and DeepSeek. All responses were rated safe by all four raters. Readability performance scores of the models achieved significant differences on FRES ( p = 0.043) and FKGL ( p < 0.001) measures. The average FKGL score (10.85–13.93) was higher than the grade scores recommended for patient education information. There was heterogeneity in the platforms ability to report on aspects of transparency, information quality and clinical accuracy while consistently failing on readability. Patient comprehension was not evaluated.