Comparative evaluation of three large language model systems and two urologists in the clinical management of benign prostatic hyperplasia and lower urinary tract symptoms: a blinded, case-based study

Large language models (LLMs) are increasingly being used to generate clinical recommendations; however, their performance in the guideline-based management of benign prostatic hyperplasia and lower urinary tract symptoms (BPH/LUTS) remains uncertain. We compared responses from three selected LLM systems with those of two participating urologists and examined whether blinded evaluators could distinguish between AI-generated and physician-generated responses. Forty synthetic BPH/LUTS cases were addressed by three LLM systems (designated as GPT-5.5, Claude, and Gemini) and two urologists, yielding a total of 200 responses. Two urologists, acting as blinded evaluators, independently scored each response using a five-dimensional rating scale (0–10), classified the source of the response (AI or human), and assessed potential harms. The reviewed analyses were reconstructed based on the original evaluator records. Data from a third backup evaluator were excluded from the analysis because the independence of the scoring could not be verified. Mean response-level scores across sources ranged from 8.79 to 8.88. No significant source effect was detected (F(4,156) = 0.29; p = 0.883), and the pooled AI–urologist difference was + 0.02 points (95% CI: -0.12 to 0.16). Inter-rater reliability was very low (quadratic-weighted κ: -0.081 to 0.087; ICC(A,1) = 0.014); this significantly limited the interpretation of the results. Results remained similar even after excluding six cases sensitive to EAU 2026 guidelines (F(4,132) = 0.57; p = 0.688). Source identification accuracy was 53.8% (215/400; case-cluster bootstrap 95% CI: 48.5%–59.0%), with a balanced accuracy of 0.532 and low inter-rater agreement (κ = 0.088). Four responses (2.0%) were flagged as potentially harmful by at least one rater; however, no response was flagged as such by both raters. No significant difference in mean evaluation scores was found between the sampled LLM systems and urologists; however, the very low inter-rater reliability precludes interpreting this finding as evidence of equivalence or broader comparative performance. Source attribution demonstrated limited and inconsistent discriminative ability, while safety findings were sparse and exploratory in nature. Not applicable. This study did not evaluate a health intervention in human patients; instead, it compared text-based clinical recommendations against a reference standard.