Readability, Quality, Understandability, and Actionability of ChatGPT Generated GI Patient Education Versus AGA Patient Center

BACKGROUND AND AIMS: Patients increasingly use the internet and artificial intelligence chatbots to obtain health information, yet the readability, quality, understandability, and actionability of AI-generated gastrointestinal patient education remain unclear. This study compared gastrointestinal patient education from a professional society website with content generated by ChatGPT using validated health literacy instruments. METHODS: In this cross-sectional comparative study, 50 gastrointestinal patient education topics from the American Gastroenterological Association patient information website were paired with ChatGPT-generated responses using standardized prompts. Readability was assessed using the Flesch-Kincaid Grade Level, quality of treatment information was evaluated using the DISCERN instrument, and understandability and actionability were assessed using the Patient Education Materials Assessment Tool; scoring was performed by two blinded reviewers. Paired t tests were used to compare mean scores between sources, and intraclass correlation coefficients (ICCs) were used to assess interrater reliability between reviewers. RESULTS: Fifty paired topics were analyzed. The mean Flesch-Kincaid Grade Level was higher for ChatGPT than GI website materials (10.33 ± 1.5 vs 8.72 ± 1.7; mean difference, 1.61; P < .001). Differences in DISCERN scores (63.5 ± 5.7 vs 64.3 ± 5.4; mean difference, - 0.8), PEMAT understandability (87.9% ± 6.9% vs 86.5% ± 7.8%; mean difference, 1.4%; P = .33), and PEMAT actionability (78.6% ± 9.8% vs 77.9% ± 10.2%; mean difference, 0.6%; P = .73) were not statistically significant. Inter-rater reliability was excellent across all measures, with intraclass correlation coefficients of 0.97 (95% CI, 0.95-0.99) for PEMAT understandability, 0.96 (95% CI, 0.94-0.98) for PEMAT actionability, and 0.99 (95% CI, 0.98-0.99) for DISCERN. CONCLUSION: ChatGPT-generated gastrointestinal patient education demonstrated similar quality, understandability, and actionability compared with professional society materials but was written at a significantly higher reading level. Improving readability may enhance accessibility and support the safe integration of AI-generated patient education.