Evaluation of large language models for clinical decision support in prostate cancer: a multidisciplinary Delphi study

Aim: Large language models (LLMs) may help interpret clinical guideline content. However, their performance in guideline-based prostate cancer (PC) scenarios remains insufficiently validated. This study evaluated the agreement between a multidisciplinary expert panel and responses generated by three leading LLMs (Claude, Gemini and ChatGPT) when addressing structured, guideline-derived clinical questions on PC. Materials & methods: A multidisciplinary tumor board created 25 structured questions based on the 2025 European Association of Urology guidelines. The guideline figures and tables were transformed into plain-language summaries for use by the LLMs. Each model was prompted with a simple, unmodified query and a panel of experts in PC management independently rated their level of agreement with the responses. The evaluation used a two-round modified Delphi method with predefined consensus criteria. Results: Eleven experts evaluated 25 guideline-based questions. After two Delphi rounds, consensus in agreement was reached for 23/25 responses (92%) with Gemini, 20/25 (80%) with ChatGPT and 14/25 (56%) with Claude (p = 0.0033). Explicit disagreement was noted in three questions (12%) with Claude after the second round. Median scores of at least 7 out of 9, indicating strong agreement, were achieved in 24/25 responses (96%) for Gemini, 22/25 (88%) for ChatGPT, and 14/25 (56%) for Claude (p = 0.0002). Most consensus difficulties involved complex cases, advanced imaging, genetic testing or questions with limited data or controversial topics. Conclusion: Advanced LLMs, when provided with PC guidelines with plain-language adaptations of figures and tables, can achieve strong alignment with expert consensus in routine situations. However, performance varies among models, with Claude showing lower agreement rates, especially in complex cases. These results emphasize the importance of expert oversight, continuous validation and standardized guideline formats interpretable by both clinicians and AI systems.