Comparison of expert-rated quality and readability of four large language models answering patient-oriented questions on Peyronie’s disease

Large language models (LLMs) are increasingly used by patients seeking information about medical conditions, including Peyronie’s disease (PD). However, the quality and readability of artificial intelligence (AI)-generated responses to patient-oriented PD questions remain unclear. To compare the responses generated by four widely used AI models for common patient-oriented questions about PD in terms of expert-rated quality, response length, and readability. In this observational study, seven patient-oriented questions on PD were generated based on Google Trends outputs. Identical English-language prompts were submitted once to four publicly available AI systems: ChatGPT powered by GPT-5.5 Instant, Gemini 3.5 Flash, DeepSeek-V4-Flash (Instant mode), and Grok (Auto mode). The resulting responses were independently evaluated in randomized order by three board-certified urologists using a predefined 4-point expert-rating scale. Response length was assessed using word count, while readability was evaluated using the Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Score (GFS), Simple Measure of Gobbledygook (SMOG), and Readable Rating. Given the small number of paired observations, model comparisons were performed using exact Friedman tests; significant overall tests were followed by exact two-sided Wilcoxon signed-rank tests with Bonferroni correction. Reviewer-specific exact Friedman analyses did not detect statistically significant differences in expert-rated quality among the four AI models (all exact p ≥ 0.111). Inter-rater agreement was limited (single-measure ICC = 0.063, 95% CI, – 0.115 to 0.305), restricting the interpretation of comparative quality findings. Exact Friedman analyses demonstrated significant overall differences in word count, FRES, FKGL, GFS, SMOG, and Readable Rating (all exact p < 0.001). However, no pairwise comparison reached the Bonferroni-corrected significance threshold, and these analyses were severely constrained by the small number of paired observations; their non-significance should therefore not be interpreted as evidence that between-model differences were absent. Commonly used AI models showed significant overall differences in response length and readability when answering patient-oriented questions about PD, whereas reviewer-specific analyses did not detect statistically significant differences in expert-rated quality. However, the limited inter-rater agreement and the absence of external reference-standard verification preclude conclusions regarding equivalent or comparable model quality and warrant cautious interpretation of the expert-rated quality findings.