Quality assessment of DeepSeek-R1’s responses to guideline-related questions on hip osteoarthritis from AAOS

Background: This study aims to evaluate the quality of responses generated by DeepSeek to guideline-related questions on hip osteoarthritis from the American Academy of Orthopaedic Surgeons (AAOS). Materials and Methods: Seventeen open-ended questions were derived from the 2023 AAOS Clinical Practice Guidelines for hip osteoarthritis. Four prompt types were applied to DeepSeek-V3 and R1. Responses were evaluated for accuracy, readability, and the mention of guidelines. Automatic evaluation and qualitative evaluation jointly evaluated the chain-of-thought of the R1 group. Results: All the models showed superior response quality. Automatic and qualitative evaluations showed that the R1 group models performed better in the BERTScore and ROUGE-L metrics, indicating that their semantics and logic were closer to the reference text. This showed that the models in the R1 group could reflect chain-of-thought reasoning to a certain extent. The flesch-kincaid reading ease score of group R1 was significantly higher than that of group V3 (P<0.001). The flesch-kincaid grade level score also showed that the R1 group was significantly better than the V3 group (P<0.001). However, the R1 group still required reading levels at the level of a 12th-grade or a college freshman. In addition, the guideline mention rate was higher in group R1 than in group V3. Conclusion: DeepSeek-R1 demonstrated high accuracy in answering open-ended questions regarding hip osteoarthritis while effectively showcasing its chain-of-thought reasoning capabilities. Moreover, R1 based on background texts showed better readability, although its readability still required a 12th-grade or freshman reading level.