Answers are grounded in this paper and cite their sources.
Background Large language models (LLMs) are increasingly used for diagnosis and clinical decision-making; however, their role in Traditional Chinese Medicine (TCM)—with its distinct theoretical framework, remains undetermined. This gap is compounded by growing worldwide interest in TCM and a shortage of highly trained experts. Objective This exploratory single-case benchmarking study evaluated how publicly available LLMs performed on diagnostic reasoning and herbal prescribing tasks for one standardized migraine case, compared with two study-elicited TCM physician responses and one published case-reference response. Methods Nine LLMs (eight reasoning-enabled “thinking” models and one non-thinking baseline) were prompted through publicly available interfaces using a structured five-task format (Western diagnosis, TCM diagnosis, treatment principle, herbal prescription, and preventive care). No fine-tuning or external knowledge bases were applied. For comparison, two experienced TCM physicians independently generated responses using the same structured format. The original published case-report response was retained as a separate reference response, not as a fully equivalent study-elicited physician comparator. Thirty-two expert raters, blinded to response source, independently scored all outputs across five evaluation dimensions. Results In this single standardized migraine case, several LLMs, including GPT o3, Qwen 3, and Gemini 2.5 Pro, received expert ratings comparable to or higher than selected comparator responses in some domains. Aya Expanse 32B received lower ratings; however, this descriptive comparison should not be interpreted as isolating the effect of reasoning mode. Conclusions These findings provide an exploratory single-case benchmark of LLM performance in a structured TCM migraine reasoning task. They should not be generalized to broader TCM practice, other diseases, or real-world clinical care without multi-case validation, stronger controls for training-data contamination, and broader physician comparator groups.
No takes yet. Share an insight, caveat, or question.
Li et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: