Key result
GPT-4 outperforms GPT-3.5 on anesthesiology exams with ~68% accuracy, scoring higher on English translations.
Why the study?
ChatGPT performance in anesthesiology had been tested using English examinations, but its effectiveness in the Korean language context remained unexplored.
Does GPT-4 outperform GPT-3.5 and CLOVA X on Korean anesthesiology in-training examination questions?
Observational (n=398)
Does GPT-4 outperform GPT-3.5 and CLOVA X on Korean anesthesiology in-training examination questions?
Absolute Event Rate: 67.8% vs 37.2%
GPT-4 outperforms GPT-3.5 and CLOVA X on Korean anesthesiology in-training exams, but its performance is significantly better when questions are translated into English, highlighting language processing discrepancies.
BACKGROUND ChatGPT has been tested in health care, including the US Medical Licensing Examination and specialty exams, showing near-passing results. Its performance in the field of anesthesiology has been assessed using English board examination questions; however, its effectiveness in Korea remains unexplored. OBJECTIVE This study investigated the problem-solving performance of ChatGPT in the fields of anesthesiology and pain medicine in the Korean language context, highlighted advancements in artificial intelligence (AI), and explored its potential applications in medical education. METHODS We investigated the performance (number of correct answers/number of questions) of GPT-4, GPT-3.5, and CLOVA X in the fields of anesthesiology and pain medicine, using in-training examinations that have been administered to Korean anesthesiology residents over the past 5 years, with an annual composition of 100 questions. Questions containing images, diagrams, or photographs were excluded from the analysis. Furthermore, to assess the performance differences of the GPT across different languages, we conducted a comparative analysis of the GPT-4’s problem-solving proficiency using both the original Korean texts and their English translations. RESULTS A total of 398 questions were analyzed. GPT-4 (67.8%) demonstrated a significantly better overall performance than GPT-3.5 (37.2%) and CLOVA-X (36.7%). However, GPT-3.5 and CLOVA X did not show significant differences in their overall performance. Additionally, the GPT-4 showed superior performance on questions translated into English, indicating a language processing discrepancy (English: 75.4% vs Korean: 67.8%; difference 7.5%; 95% CI 3.1%-11.9%; P=.001). CONCLUSIONS This study underscores the potential of AI tools, such as ChatGPT, in medical education and practice but emphasizes the need for cautious application and further refinement, especially in non-English medical contexts. The findings suggest that although AI advancements are promising, they require careful evaluation and development to ensure acceptable performance across diverse linguistic and professional settings.
No takes yet. Share an insight, caveat, or question.
Yoon et al. (2024) conducted an observational in Anesthesiology and pain medicine in-training examination (n=398). GPT-4 vs. GPT-3.5 and CLOVA X was evaluated on Performance (number of correct answers/number of questions). GPT-4 achieved 67.8% correct answers on Korean anesthesiology in-training exams, outperforming GPT-3.5 (37.2%) and CLOVA-X (36.7%), and scored higher on English-translated questions (75.4%, P=.001).
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: