Answers are grounded in this paper and cite their sources.
Aim Large language models (LLMs) are increasingly employed to acquire medical information. However, their reliability in addressing rare and technically intricate conditions, such as female urethral stricture disease (FUSD), remains uncertain. This study evaluated the guideline concordance, information quality, reliability, and readability of responses generated by three widely accessible LLMs in response to clinical questions related to FUSD. Methods This comparative cross-sectional study was conducted in January 2026. Thirty standardized questions across five clinical domains were developed based on the 2025 European Association of Urology Guidelines for Urethral Stricture Disease and refined through expert consensus. These questions were submitted to ChatGPT 5.0, Gemini 3.0, and Microsoft Copilot (Basic) under identical conditions. Responses were independently evaluated by three blinded board-certified urologists using a 5-point expert-rated accuracy scale. Information reliability and quality were assessed using modified DISCERN and ensuring quality information for patients (EQIP) independently by the blinded reviewers, while readability was evaluated using flesch reading ease score (FRES) and flesch-kincaid grade level (FKGL). Between-model comparisons were performed using the Friedman test, followed by Wilcoxon signed-rank tests with Bonferroni correction. Results Gemini achieved the highest expert-rated accuracy (4.80±0.41) and modified DISCERN score (19.0±1.7) among the three models. The Copilot achieved the highest EQIP (89.60±4.01) and FRES (42.23±3.89) scores and the lowest FKGL (12.67±1.04). Significant differences between the models were identified across all evaluated metrics. Gemini had significantly higher modified DISCERN scores than ChatGPT and Copilot (both p<0.001), whereas ChatGPT and Copilot did not differ significantly. Conclusion The evaluated LLMs demonstrated distinct performance profiles rather than uniform superiority. Gemini showed the highest expert-rated accuracy and information reliability, whereas Copilot provided the most favorable patient-oriented information quality and readability. Large language model-generated information on FUSD should be verified against current clinical guidance before clinical or patient education.
No takes yet. Share an insight, caveat, or question.
Şevik et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: