ChatGPT is one of the most advanced large language models. We aim to examine ChatGPT-4o accuracy in solving radiobiology computational problems. Two board-certified radiation oncologists created a problem set consisting of 30 questions. We used OpenAI API to query the ChatGPT(model: GPT-4o) and generate corresponding answers. Answers were graded using a 3-score system. We conducted subgroup analysis for no prompts(zero-shot learning) and different prompts, questions with or without alpha-beta ratio, and question categories. ChatGPT correctly answered approximately 60% of questions without any prompting strategies. While ChatGPT demonstrates stable performance in structured calculation problems, particularly those involving alpha-beta ratios, it still exhibits notable limitations in handling multi-step reasoning and clinical decision-making tasks. This result highlights the need for integrating professional tools and refining prompting strategies to enhance their practical utility.
Su et al. (Thu,) studied this question.