Quality, understandability, and actionability of large language model responses to fecal incontinence: a cross-sectional study

Background Patients increasingly use large language models (LLMs) for health information, yet the evaluation of the quality of fecal incontinence (FI) content from these models is limited. We aimed to evaluate the quality, understandability, and actionability of LLM responses to FI queries. We hypothesized that LLMs would provide variable quality responses with variable understandability and actionability. Methods Search data was extracted from Google Trends’ top five search queries related to FI to create identical prompts used to generate responses from four LLMs: ChatGPT version 4o mini (OpenAI), Perplexity version sonar (Perplexity AI), LLaMA version 3.2 (Meta AI), and Gemini version 2.0 flash (Google). Responses were assessed using DISCERN, Patient Education Materials Assessment Tool (PEMAT), and the Flesch–Kincaid Grade Level (FKGL). Statistical analysis was performed using R version 4.4.2 (R Core Team, 2025). Results Response quality was moderate (median DISCERN 44 out of 80), nominal understandability was median 77% ([7/9] PEMAT) although inter-rater agreement was poor, and actionability was poor (median 40% [2/5] PEMAT). DISCERN scores varied significantly ( p = 0.033). The responses were at the high school to college level using the FKGL score. Readability differed significantly ( p = 0.022). Conclusions LLM responses demonstrated structural clarity using PEMAT criteria but lacked actionability and were above the recommended reading level for consumer health information. Continued evaluation of artificial intelligence is essential before LLMs can be recommended for patient education.