Background—Chatbots are increasingly used in clinical medicine. However, all studies that examined the applications of chatbots so far were on their usefulness in western medicine. There has been no study that explored their knowledge on alternative medical practices. Objectives—To examine the performance of five free chatbots (ChatGPT, Claude, Copilot, Gemini and Perplexity) as well as the subscribed versions of ChatGPT, Claude and Perplexity in answering questions on oriental medicine-based theory and practice of acupuncture and western medicine-based acupuncture-transmitted infections. Methods—A total of 2528 multiple choices and true/false questions from four books on theory and practice of acupuncture, and 100 questions on acupuncture-transmitted infections set by two microbiology and infectious disease professors, were used for evaluation of the chatbots. Results—Overall, the median score for the eight free/subscribed chatbots in answering questions on theory and practice of acupuncture (75%) was significantly lower than that on acupuncture-transmitted infections (86%) (p < 0.001). Further analysis also showed significantly lower median score for all the five free chatbots and all the three subscribed chatbots in answering the questions on theory and practice of acupuncture (75% for both) than those on acupuncture-transmitted infections (86% and 90% respectively) (p = 0.014 and p = 0.003 respectively). For the three subscribed chatbots, GPT-4o achieved the highest median score (84%), followed by Perplexity Pro (79%) and Claude 3 Opus (66%) for the questions on theory and practice of acupuncture (p = 0.036 by Kruskal Wallis test). Post-hoc Dunn’s test revealed that the median score for GPT-4o was significantly higher than Claude 3 Opus (p = 0.032). Conclusions—Performance of the chatbots on oriental medicine-based theory and practice of acupuncture was inferior to that on western medicine-based acupuncture-transmitted infections. In order to further improve their usefulness, efforts should be spent on improving the chatbots’ knowledge on the theory and practice of acupuncture.
Lee et al. (Mon,) studied this question.