Key result
Experienced surgeons outperform AI by ~45% in spinal surgery accuracy, driven by case-based clinical scenarios.
Why the study?
Spinal surgery involves high intra-operative challenges and elevated complication rates, prompting evaluation of whether AI tools can effectively assist in patient assessment and clinical decision-making compared with orthopedic surgery specialists.
Do AI models perform as well as experienced orthopedic surgeons in answering case-based and knowledge-based spinal surgery questions?
Cross-Sectional (n=13)
Do AI models perform as well as experienced orthopedic surgeons in answering case-based and knowledge-based spinal surgery questions?
AI models perform comparably to experienced orthopedic surgeons on theoretical knowledge questions but are significantly inferior in clinical judgment and case-based scenarios in spinal surgery.
Answers are grounded in this paper and cite their sources.
Current AI models warrant caution in spinal surgery decisions; leaves open potential for augmentation after prospective validation.
Objective: Spinal surgery (SS) is an area characterized by high intra-operative challenges and higher complication rates compared to several other surgical specialties. The purpose of this study is to evaluate the effectiveness of artificial intelligence (AI) instruments-Chat Generative Pre-trained Transformer (ChatGPT)-4o, DeepSeek-V3, and Gemini Pro-in patient assessment and the clinical decision-making process compared with specialists of orthopedic surgery on a series of case-based and knowledge-based questions relevant to SS. Materials and Methods: By two experienced orthopedic surgeons, a set of 50 questions has been created, including 25 requiring clinical judgement through the use of a case presentation format and 25 to test theoretical understanding. The test was given to two groups: Group 1 included three AI software programs (ChatGPT-4.0, DeepSeek-V3, Gemini Pro) and Group 2 included ten experienced orthopedic surgeons. The answers given were scored independently by the two expert surgeons. Results: Group 2 performed significantly better than Group 1 in the case-based questions. There was a significant difference between the groups in one section (p=0.025), while there was no significant difference for the knowledge-based questions section (p=1.000). On the assessment of total correct responses, Group 2’s performance was significantly better (p=0.036). Conclusion: AI technologies have proved their utility for knowledge-based tasks but are dramatically inferior to clinicians for areas requiring clinical judgement and case analysis. Even if AI algorithms can become auxiliary tools, they should not take the clinician’s place as the decision-maker.
No takes yet. Share an insight, caveat, or question.
Demir et al. (2025) conducted a cross-sectional in Spinal surgery clinical evaluation (n=13). AI models (ChatGPT-4o, DeepSeek-V3, Gemini Pro) vs. Experienced orthopedic surgeons was evaluated on Overall accuracy on a 50-question spinal surgery assessment (p=0.036). Experienced orthopedic surgeons outperformed AI models in overall accuracy on a spinal surgery assessment, particularly in case-based clinical scenarios.
Synapse has enriched 10 related papers. Consider them for comparative context: