Background/Objectives: Large language models (LLMs) are increasingly used by the public to seek health information, yet their accuracy in addressing common vaccine myths remains unclear. Sycophantic LLM behavior, where models align with rather than correct user-stated beliefs, poses specific risks in health contexts. Methods: We conducted an exploratory multi-vendor evaluation of three LLMs (GPT-5, Gemini 2.5 Flash, Claude Sonnet 4) using officially curated vaccination myths from Germany’s public health institution and two realistic user framings (curious skeptic, convinced believer). All model responses were independently evaluated by two blinded medical experts for misconception addressal (binary criterion applied to the response text), scientific accuracy, and communication clarity (5-point Likert scales). Additionally, blinded marketing experts ranked models for lay communication clarity. Flesch Reading Ease scores were computed for all outputs. Results: Across all myths, framings, and models (66 response items), both medical raters judged that all responses refuted the targeted misconception; no response affirmed or ignored a myth, including under the adversarial convinced believer framing. Scientific accuracy and clarity ratings were high and tightly clustered (median 4.0–4.5), with no combined score below 3 and substantial inter-rater agreement. Marketing experts independently ranked Gemini 2.5 Flash and GPT-5 highest for lay clarity. Readability analysis revealed generally low accessibility, particularly for the convinced believer framing and for Claude Sonnet 4 outputs. Conclusions: Our findings suggest that general-purpose LLMs can produce scientifically accurate, on-topic rebuttals to widely documented vaccine myths under realistic default conditions, although linguistic complexity and framing-sensitive style may limit accessibility. Whether such outputs change beliefs or behavior in hesitant individuals was not tested. With readability optimization, these outputs could serve as building blocks for myth-debunking tools, given prospective evaluation with behavioral endpoints.
Reis et al. (Fri,) studied this question.