Key result
GPT-3.5 achieves ~90% accuracy for appendicitis surgical decisions but shows no advantage over machine learning.
Why the study?
Robust, accessible, externally validated clinical decision support systems are sparse, and it was unknown whether GPT-3.5 could match machine learning and board-certified surgeons in appendicitis treatment decisions.
Does GPT-3.5 provide similar clinical decisions to a machine learning model and board-certified surgeons for appendectomy versus conservative treatment in patients with right abdominal pain?
Cohort (n=113)
Yes
Does GPT-3.5 provide similar clinical decisions to a machine learning model and board-certified surgeons for appendectomy versus conservative treatment in patients with right abdominal pain?
GPT-3.5 demonstrated high accuracy and agreement with board-certified surgeons for surgical decision-making in suspected appendicitis, performing comparably to a dedicated machine learning model.
Caution against routine use in appendicitis triage; hypothesis-generating for LLM-assisted surgical decisions pending validation.
Background: Nonsurgical treatment of uncomplicated appendicitis is a reasonable option in many cases despite the sparsity of robust, easy access, externally validated, and multimodally informed clinical decision support systems (CDSSs). Developed by OpenAI, the Generative Pre-trained Transformer 3.5 model (GPT-3) may provide enhanced decision support for surgeons in less certain appendicitis cases or those posing a higher risk for (relative) operative contra-indications. Our objective was to determine whether GPT-3.5, when provided high-throughput clinical, laboratory, and radiological text-based information, will come to clinical decisions similar to those of a machine learning model and a board-certified surgeon (reference standard) in decision-making for appendectomy versus conservative treatment. Methods: In this cohort study, we randomly collected patients presenting at the emergency department (ED) of two German hospitals (GFO, Troisdorf, and University Hospital Cologne) with right abdominal pain between October 2022 and October 2023. Statistical analysis was performed using R, version 3.6.2, on RStudio, version 2023.03.0 + 386. Overall agreement between the GPT-3.5 output and the reference standard was assessed by means of inter-observer kappa values as well as accuracy, sensitivity, specificity, and positive and negative predictive values with the “Caret” and “irr” packages. Statistical significance was defined as p < 0.05. Results: There was agreement between the surgeon’s decision and GPT-3.5 in 102 of 113 cases, and all cases where the surgeon decided upon conservative treatment were correctly classified by GPT-3.5. The estimated model training accuracy was 83.3% (95% CI: 74.0, 90.4), while the validation accuracy for the model was 87.0% (95% CI: 66.4, 97.2). This is in comparison to the GPT-3.5 accuracy of 90.3% (95% CI: 83.2, 95.0), which did not perform significantly better in comparison to the machine learning model (p = 0.21). Conclusions: This study, the first study of the “intended use” of GPT-3.5 for surgical treatment to our knowledge, comparing surgical decision-making versus an algorithm found a high degree of agreement between board-certified surgeons and GPT-3.5 for surgical decision-making in patients presenting to the emergency department with lower abdominal pain.
No takes yet. Share an insight, caveat, or question.
Sanduleanu et al. (2024) conducted a cohort in Suspected appendicitis (n=113). GPT-3.5 vs. Machine learning model was evaluated on Accuracy of decision-making for appendectomy versus conservative treatment (p=0.21). GPT-3.5 achieved 90.3% accuracy in surgical decision-making for suspected appendicitis, showing high agreement with surgeons and no significant difference vs a machine learning model (p=0.21).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: