Comparative analysis of five large language models for generating heart failure patient education content: dual perspectives from professionals and patients

Self-management in heart failure (HF) is essential for improving outcomes, yet adherence remains suboptimal, particularly among older adults and individuals with limited health literacy. Large language models (LLMs) generating patient education content at scale, yet evidence regarding their accuracy and alignment with HF patient needs is limited. To compare the quality of responses generated by five widely used LLMs (DeepSeek, Doubao, Grok, Gemini, and ChatGPT) to self-management questions of HF patients from two perspectives (professional and patient perception), and to quantify the discrepancies between these perspectives. A cross-sectional study. A structured 45-item question bank on self-management among HF patients, encompassing six domains and varying in type and difficulty, was administered to five LLMs. Responses were independently evaluated by four healthcare professionals using four domains (accuracy, completeness, safety, and actionability; 1–5 Likert scale with safety veto). Fifteen HF patients assessed responses in terms of comprehensibility, usefulness, satisfaction, and trustworthiness. Generalized estimating equations (GEE), intraclass correlation coefficients (ICC), and linear mixed-effects models were used to compare performance and assess variability across raters and models. DeepSeek achieved the highest professional evaluation score (19.49/20), significantly exceeding those of the other models (GEE β = −1.79 to − 1.06, all P < 0.001). In contrast, patients rated Doubao highest (16.93), followed by DeepSeek (16.33), particularly among older and lower-education subgroups. The professional–patient discrepancy varied substantially across models, ranging from 1.50 (Doubao) to 5.02 (ChatGPT). Notably, responses involving dietary sodium restriction showed consistent safety concerns across models, yet patients consistently rated these items favorably. Diet and sodium restriction consistently emerged as the weakest-performing domain across all models. Professional and patient evaluations of LLM-generated self-management education for HF patients diverged significantly. DeepSeek ranked highest clinically, but patients favored Doubao for its readability and trustworthiness, emphasizing the need to balance clinical accuracy with patient-centered design in healthcare AI.