Evaluation of Large Language Models in Thailand’s National Medical Licensing Examination

Abstract Advanced general-purpose Large Language Models (LLMs), including OpenAI’s Chat Generative Pre-trained Transformer (ChatGPT), Google’s Gemini and Anthropic’s Claude, have demonstrated capabilities in answering clinical questions, including those with image inputs. The Thai National Medical Licensing Examination (ThaiNLE) lacks publicly accessible specialist-confirmed study materials. This study aims to evaluate whether LLMs can accurately answer Step 1 of the ThaiNLE, a test similar to Step 1 of the United States Medical Licensing Examination (USMLE). We utilized a mock examination dataset comprising 300 multiple-choice questions, 10.2% of which included images. LLMs capable of processing both image and text data were used, namely GPT-4, Claude 3 Opus and Gemini 1.0 Pro. Five runs of each model were conducted through their application programming interface (API), with the performance assessed based on mean accuracy. Our findings indicate that all tested models surpassed the passing score, with the top performers achieving scores more than two standard deviations above the national average. Notably, the highest-scoring model achieved an accuracy of 88.9%. The models demonstrated robust performance across all topics, with consistent accuracy in both text-only and image-enhanced questions. However, while the LLMs showed strong proficiency in handling visual information, their performance on text-only questions was slightly superior. This study underscores the potential of LLMs in medical education, particularly in accurately interpreting and responding to a diverse array of exam questions.