Answers are grounded in this paper and cite their sources.
Millions consult AI chatbots and LLMs for health advice daily, yet systematic auditing methods for assessing the quality of AI responses beyond their informational accuracy remain scarce. This study introduces an automated NLP pipeline generating N = 1,320 simulated health conversations across 11 domains, three model tiers (GPT-4o, GPT-5.2-chat, GPT-5.2-thinking), and two prompt framing conditions (neutral vs. health influencer framed AI prompts). Applying VADER sentiment, Flesch readability, and zero-shot bias detection, a paradox of progress emerged: more capable model tiers produced longer, more readable yet colder and more biased responses across all six bias dimensions, while the free model tiers delivered paradoxically warmer responses with the lowest detected bias. Health influencer framing in AI prompts silently degraded accessibility across all model tiers. Not one of 1,320 responses met recommended reading levels for patient-facing health materials. Quality gaps are real, multidimensional, non-linear, and invisible to users. The pipeline and conceptual framework introduced here offer a scalable, reproducible tool for AI communication audits.
No takes yet. Share an insight, caveat, or question.
Sebastian Scherr (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: