Key result
Human expert materials outperform March 2026-generation LLMs in overall quality for HF self-management education.
Why the study?
Do March 2026-generation large language models provide comparable information quality, readability, actionability, and clinical accuracy to expert-written materials for heart failure self-management education?
Cross-Sectional (n=32)
Fully blinded
Do March 2026-generation large language models provide comparable information quality, readability, actionability, and clinical accuracy to expert-written materials for heart failure self-management education?
p-value: p=<0.001
While March 2026-generation LLMs show promise for heart failure patient education, they still generate high-risk clinical errors and require mandatory clinical review before patient use.
Expert materials should remain the standard for heart failure self-management education; leaves open whether future LLMs can match quality and safety.
Self-management education is a cornerstone of chronic heart failure (HF) care, yet clinical workload limits its delivery. Patients increasingly use large language models (LLMs) for HF information, but comparative data on March 2026-generation models remain scarce. We compared five publicly available LLMs against expert-written HF education materials across information quality, readability, actionability, clinical accuracy, and safety. Thirty-two standardized patient questions across eight HF self-management domains were submitted in zero-shot format to five LLMs (Claude Sonnet 4.6, GPT-5.2, DeepSeek-V3.2, Kimi K2.5, Gemini 3.1 Pro). Six fully blinded raters evaluated anonymized responses using DISCERN, PEMAT-P, four readability indices (FKGL, GFI, SMOG, ARI), clinical accuracy ratings, and a three-tier error severity scale. Inter-rater reliability (ICC) and linear mixed-effects models were used to compare performance across sources, topics, and their interaction. Inter-rater reliability was good (ICC 0.77–0.84, all p < 0.001). Human expert materials outperformed all LLMs across all metrics (all adjusted p < 0.05). Among LLMs, GPT-5.2 achieved the highest estimated DISCERN scores (68.7, 95% CI 67.4–70.0), Claude Sonnet 4.6 the highest clinical accuracy (92.4%, 95% CI 91.3–93.5%), DeepSeek-V3.2 the highest PEMAT actionability (82.4, 95% CI 80.9–83.9), and Kimi K2.5 the most favorable ARI readability (7.9, 95% CI 7.5–8.3). However, pairwise differences between top-performing LLMs did not always reach statistical significance, and topic-level estimates were based on only four questions per domain. Significant source × topic interactions were observed (all p < 0.001). Total clinical errors ranged from 8 (Claude Sonnet 4.6) to 27 (Gemini 3.1 Pro); high-risk errors were distributed across all five LLMs, concentrated in warning-sign and medication domains. March 2026-generation LLMs show domain-specific promise for HF patient education, but high-risk errors persist across all systems. LLM-generated content requires mandatory clinical review, and model selection should match specific educational tasks rather than relying on generic performance rankings.
No takes yet. Share an insight, caveat, or question.
Jiang et al. (2026) conducted a cross-sectional in Heart failure (n=32). March 2026-generation large language models vs. Expert-written heart failure education materials was evaluated on Overall quality score (1-5 Likert scale) (p=<0.001). Human expert materials significantly outperformed all evaluated March 2026-generation large language models in overall quality for heart failure self-management education.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: