Artificial intelligence chatbots as information sources for herpes zoster: an evaluation of patient-education suitability, overall quality, and readability

Objective Clarify the effects of large-language-model types and health-education content categories on generated-text quality (patient-education suitability, overall quality) and readability, so as to provide empirical evidence for standardized AI-assisted health-communication applications. Methods This cross-sectional study was conducted between July 10 and July 15, 2026. Twenty herpes zoster-related questions were developed by a dermatologist and a pharmacist and categorized into five themes. The questions were then entered into five mainstream large language models (GPT-5, Doubao, DeepSeek, Qianwen, and Gemini), generating 100 textual responses. Response quality was assessed using the Chinese version of the Patient Education Materials Assessment Tool for Printable Materials (C-PEMAT-P) and Global Quality Score (GQS). Readability was evaluated via seven indices including the Automated Readability Index (ARI) and Flesch Reading Ease Score (FRES). Readability outcomes were compared using one-way repeated-measures analysis of variance with Greenhouse–Geisser correction. C-PEMAT-P raw totals and ordinal GQS scores were compared using Friedman tests. Post-hoc comparisons were adjusted using the Holm method. Category-level comparisons were performed using linear mixed-effects models with likelihood-ratio χ 2 tests for continuous outcomes and an ordinal generalized estimating equation with a Wald χ 2 test for GQS. Results Significant overall differences among models were observed for all seven readability indices and both assessment scores (all p < 0.001). Median C-PEMAT-P raw total scores were 12.50, 11.00, 10.00, 9.00 and 9.00 for GPT-5, Doubao, DeepSeek, Qianwen and Gemini, respectively; corresponding median GQS scores were 5.00, 4.00, 4.00, 3.00, and 3.00. After Holm correction, GPT-5 achieved higher scores than each of the other models on both assessment measures. However, no significant difference in C-PEMAT-P raw totals was found between DeepSeek and either Qianwen or Gemini, or between Doubao and DeepSeek, after adjustment (all adjusted p = 0.062), and Qianwen and Gemini did not differ significantly (adjusted p = 0.766). The overall effect sizes were Kendall’s W = 0.611 for C-PEMAT-P and W = 0.779 for GQS. Exploratory pooled Spearman correlation analysis revealed a positive association between the two assessment scores (ρ = 0.853). Conclusion Model type is a key determinant of text quality. Readability was independently influenced by both model type and content category, and the two are relatively independent. In clinical applications, GPT-5, DeepSeek, and Doubao are preferred. Readability can be optimized by integrating topic characteristics and patients’ health literacy to enhance communication accuracy. Limitations exist in this study regarding sample coverage and evaluation dimensions (the absence of subjective experience data); larger-scale verification should be conducted in future research.