Key result
ChatGPT 4 correctly answers ~73% of orthopaedic shoulder and elbow self-assessment questions, beating ChatGPT 3.5.
Why the study?
The study was conducted to evaluate the performance and evolution of ChatGPT as an information resource in shoulder and elbow surgery using self-assessment questions.
Does ChatGPT 4 improve accuracy compared to ChatGPT 3.5 on orthopedic shoulder and elbow self-assessment questions?
Does ChatGPT 4 improve accuracy compared to ChatGPT 3.5 on orthopedic shoulder and elbow self-assessment questions?
ChatGPT 4 significantly outperformed ChatGPT 3.5 on orthopedic shoulder and elbow self-assessment questions, though neither is accurate enough to replace clinical decision-making.
Answers are grounded in this paper and cite their sources.
ChatGPT 4 improves accuracy on shoulder and elbow questions; leaves open clinical use pending prospective validation.
Background The purpose of this study was to evaluate the performance and evolution of Chat Generative Pre-Trained Transformer (ChatGPT; OpenAI) as a resource for shoulder and elbow surgery information by assessing its accuracy on the American Academy of Orthopaedic Surgeons shoulder-elbow self-assessment questions. We hypothesized that both ChatGPT models would demonstrate proficiency and that there would be significant improvement with progressive iterations. Materials and Methods A total of 200 questions were selected from the 2019 and 2021 American Academy of Orthopaedic Surgeons shoulder-elbow self-assessment questions. ChatGPT 3.5 and 4 were used to evaluate all questions. Questions with non-text data were excluded (114 questions). Remaining questions were input into ChatGPT and categorized as follows: anatomy, arthroplasty, basic science, instability, miscellaneous, nonoperative, and trauma. ChatGPT's performances were quantified and compared across categories with chi-square tests. The continuing medical education credit threshold of 50% was used to determine proficiency. Statistical significance was set at P <.05. Results ChatGPT 3.5 and 4 answered 52.3% and 73.3% of the questions correctly, respectively ( P =.003). ChatGPT 3.5 performed significantly better in the instability category ( P =.037). ChatGPT 4's performance did not significantly differ across categories ( P =.841). ChatGPT 4 performed significantly better than ChatGPT 3.5 in all categories except instability and miscellaneous. Conclusion ChatGPT 3.5 and 4 exceeded the proficiency threshold. ChatGPT 4 performed better than ChatGPT 3.5, showing an increased capability to correctly answer shoulder and elbow-focused questions. Further refinement of ChatGPT's training may improve its performance and utility as a resource. Currently, ChatGPT remains unable to answer questions at a high enough accuracy to replace clinical decision-making. [ Orthopedics . 2025;48(2):e69–e74.]
No takes yet. Share an insight, caveat, or question.
Nieves-Lopez et al. (2025) studied Orthopaedic shoulder and elbow self-assessment questions (n=86). ChatGPT 4 vs. ChatGPT 3.5 was evaluated on Percentage of questions answered correctly (p=.003). ChatGPT 4 correctly answered 73.3% of orthopaedic shoulder and elbow self-assessment questions compared to 52.3% for ChatGPT 3.5, demonstrating improved proficiency.
Synapse has enriched 10 related papers. Consider them for comparative context: