What is the clinical evidence from this study?

Study design: Other. Population: Pediatric Surgery (EBPS exam questions) (n=210). Intervention: GPT-4 vs. GPT-3.5. Primary outcome: Correctly answered questions (OR 4.1, p=<0.001).

June 7, 2023Open Access

How does ChatGPT perform on the European Board of Pediatric Surgery examination? A randomized comparative study

Q: What are the key findings of this study?

GPT-4 significantly outperformed GPT-3.5 in correctly answering European Board of Pediatric Surgery exam questions, achieving an accuracy of 76.2% compared to 43.8% (OR 4.1, p<0.001).

Key Result

GPT-4 significantly outperformed GPT-3.5 in correctly answering European Board of Pediatric Surgery exam questions, achieving an accuracy of 76.2% compared to 43.8% (OR 4.1, p<0.001).

Structured PICO

Population

210 randomly selected sample questions from the European Board of Pediatric Surgery (EBPS) exam (105 questions per set).

Intervention

GPT-4 language model

Comparator

GPT-3.5 language model

Outcome

Accuracy (percentage of correct answers) on the EBPS exam questions

GPT-4 significantly outperforms GPT-3.5 in answering pediatric surgery board examination questions, suggesting improved capability for medical knowledge retrieval.

Main Result

Effect estimate: OR 4.1

Absolute Event Rate: 76.2% vs 43.8%

p-value: p=<0.001

Limitations

The ChatGPT algorithm was initially trained on a corpus constructed using data produced on or before 2021.
The model is closed and does not have a public API, preventing fine-tuning with task-specific data.
The version of ChatGPT used was an older model than the one available at the time of the study's completion.

Abstract

Abstract Purpose The purpose of this study was to conduct a detailed comparison of the accuracy and responsiveness of GPT-3.5 and GPT-4 in the realm of pediatric surgery. Specifically, we sought to assess their ability to correctly answer a series of sample questions of European Board of Pediatric Surgery (EBPS) exam. Methods This study was conducted between 20 May 2023 and 30 May 2023. This study undertook a comparative analysis of two AI language models, GPT-3.5 and GPT-4, in the field of pediatric surgery, particularly in the context of EBPS exam sample questions. Two sets of 105 (total 210) sample questions each, derived from the EBPS sample questions, were collated. Results In General Pediatric Surgery, GPT-3.5 provided correct answers for 7 questions (46.7%), and GPT-4 had a higher accuracy with 13 correct responses (86.7%) (p = 0.020). For Newborn Surgery and Pediatric Urology, GPT-3.5 correctly answered 6 questions (40.0%), and GPT-4, however, correctly answered 12 questions (80.0%) (p = 0.025). In total, GPT-3.5 correctly answered 46 questions out of 105 (43.8%), and GPT-4 showed significantly better performance, correctly answering 80 questions (76.2%) (p < 0.001). Given the total responses, when GPT-4 was compared with GPT-3.5, the Odds Ratio was found to be 4.1. This suggests that GPT-4 was 4.1 times more likely to provide a correct answer to the pediatric surgery questions compared to GPT-3.5. Conclusion This comparative study concludes that GPT-4 significantly outperforms GPT-3.5 in responding to EBPS exam questions.

Ask AI

Helpful

Bookmark

View Full Paper