Abstract Rationale Large language models (LLMs) are transforming medical communication by enabling the rapid creation of patient-centered educational content. However, their consistency, linguistic precision, and fidelity to clinical guidelines have not been systematically evaluated. This study compared the performance of four LLMs: ChatGPT-5, ChatGPT-4, Gemini, and OpenEvidence; in generating asthma education materials aligned with the 2025 Global Initiative for Asthma (GINA) guidelines. Methods This cross-sectional evaluation analyzed model outputs generated using a standardized prompt emphasizing accuracy, empathy, and SMOG 8th-grade readability. Each LLM was provided the complete GINA guideline document. Two board-certified pulmonologists independently and blindly reviewed each model’s responses across nine domains encompassing disease overview, diagnosis, management, and adherence. Disagreements were resolved by adjudication. Models were rated using a structured scoring framework assessing adequacy, comprehensiveness, readability, empathy, understandability, safety, and concordance with source guidelines. All systems operated under default configurations without temperature adjustment or fine-tuning. Inter-model comparisons of median adequacy, comprehensiveness, and readability scores were performed using the Kruskal-Wallis test. Results ChatGPT-4 achieved the most balanced performance, with the highest mean scores for adequacy (8.6), comprehensiveness (7.6), and understandability (9.0). ChatGPT-5 generated the clearest lay language (readability 8.2), Gemini produced the most empathetic tone (7.0), and OpenEvidence demonstrated perfect guideline concordance (10.0) with no hallucination or safety violations. Kruskal-Wallis analysis showed significant inter-model variation in adequacy (p = 0.02) and comprehensiveness (p = 0.03), while differences in readability were not statistically significant (p = 0.21). Conclusions ChatGPT-4 provided the most clinically coherent and contextually precise patient materials, whereas OpenEvidence achieved superior factual reliability and safety. These findings reveal distinct performance profiles across contemporary LLMs, suggesting that while some models excel in clarity or empathy, others demonstrate stronger fidelity to guidelines. Further research should investigate how temperature tuning, reinforcement strategies, and contextual depth influence accuracy, tone, and interpretability. Continued clinician oversight and structured validation remain critical to the ethical and effective integration of artificial intelligence into patient education and respiratory care. This abstract is funded by: none
Phadke et al. (Fri,) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: