Answers are grounded in this paper and cite their sources.
Objectives To compare the accuracy, empathy, and helpfulness of consumer AI chatbot responses with physician-written responses to psychosocial questions raised by people with cancer; to test whether expert raters could distinguish physicians from AI responses; and to establish the Cancer Psychosocial Response Benchmark (CPRB-15) for longitudinal model evaluation. Methods Fifteen questions spanning 5 psychosocial domains were answered by ChatGPT, Claude, Gemini, DeepSeek, and 1 of 2 psycho-oncology psychiatrists. Eight psychiatry and psycho-oncology clinicians rated randomized, source-blinded responses from 1 to 5 for accuracy, empathy, and helpfulness and attempted to identify the physician response. Primary crossed linear mixed-effects models included ChatGPT, Claude, Gemini, and physician responses; DeepSeek was descriptive because it was under-sampled. Results The study yielded 420 response-level evaluations and 1,260 metric-level ratings. Compared with physician responses, adjusted composite ratings were higher for ChatGPT (+0.76; 95% CI, +0.56 to +0.96), Claude (+0.72; +0.52 to +0.92), and Gemini (+0.32; +0.12 to +0.52). Experts identified the physician response in 23 of 105 judgments (21.9%; exact 95% CI, 14.4–31.0%), which did not differ from the prespecified 20% chance rate ( p = 0.626). Higher readability grade level correlated with higher composite rating ( r = 0.61; p < 0.001). Significance of results Current chatbots can produce concise psychosocial cancer responses that expert clinicians rate highly, but these text-only findings do not establish clinical equivalence or support autonomous use. The frozen questions, responses, ratings, and blinding judgments form CPRB-15, a reusable benchmark for tracking communication quality as models evolve.
No takes yet. Share an insight, caveat, or question.
Louis et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: