Validity and reliability of ChatGPT's responses on dietary supplements in Japan: A quality assessment and content analysis

This study evaluated the validity and reliability of large language model (LLM) responses on dietary supplements (DS), a domain marked by scientific controversy and misinformation. The goal was to support informed consumer decisions and guide improvements in LLM performance. We collected responses from GPT-4 and GPT-4o on the effects of 30 DS on six diseases. Two medical professionals categorized each response as “Effective,” “Uncertain,” or “Not Effective.” They also created a guideline to assess evidence-based effectiveness and compared it with LLM-generated responses to determine accuracy. Additionally, we conducted qualitative content analysis to identify response patterns and misleading content. GPT-4 and GPT-4o affirmed DS effectiveness in only 10% of cases, with 40% rated as “Uncertain” and 50% as “Not Effective.” Accuracy was about 57%, considerably lower than that observed in nutrition-related studies (57% in DS vs. 80% ~ in structured nutrition tasks”). Content analysis showed templated responses, frequent ambiguity, and occasional inclusion of irrelevant or incorrect information. Our findings suggest that ChatGPT's responses on dietary supplements are generally cautious but often ambiguous, with a moderate risk of misinformation. As generative AI becomes a common source for health advice, these limitations could mislead users. Enhancing LLMs' evidence-based accuracy and ensuring consistent professional guidance are essential. This is the first study to assess the validity and reliability of LLM-generated responses on dietary supplements using both quantitative and qualitative methods. We also developed a novel evidence-based framework to evaluate supplement effectiveness, providing a new tool for future research and supporting safer AI-assisted health communication. • This is the first study to assess GPT-4 and GPT-4o on dietary supplements. • Only 10% of responses supported supplements as treatments for specific diseases. • GPT responses achieved 57% accuracy, lower than in structured nutrition knowledge. • LLMs achieved highest accuracy for cancer and lowest for constipation topics. • LLMs often produce ambiguous, lengthy, or expectation-driven responses.