Evaluation of large language models for clinical decision support in penile and testicular cancer: a multidisciplinary Delphi study

Aim: Large language models (LLMs) may help interpret clinical guideline information, but their performance in penile and testicular cancer remains poorly defined. We evaluated agreement between multidisciplinary expert panels and responses generated by Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini to guideline-based questions on these cancers. Materials & methods: Specialized tumor boards developed 25 questions for each cancer based on the 2025 European Association of Urology guidelines. The original guideline text, with figures and tables converted into structured, plain-language textual descriptions, was provided to each LLM. Separate panels of nine experts evaluated the responses using a two-round modified Delphi process with predefined consensus criteria. Results: For penile cancer, consensus in agreement was reached on 21/25 responses (84%) for Claude, 24/25 (96%) for Gemini and 23/25 (92%) for ChatGPT, with no statistically significant differences in overall or pairwise comparisons. For testicular cancer, consensus in agreement was reached on 24/25 items (96%) for each model. One Claude response and one ChatGPT response reached consensus in disagreement. No significant inter-model differences were observed. In both cancers, most responses not reaching consensus or requiring a second round related to prognosis or staging. Across both cancers combined, consensus in agreement was 90% for Claude, 96% for Gemini and 94% for ChatGPT, with no significant inter-model differences. Median scores ≥7 occurred in 94–98% of responses for all LLMs. Conclusion: Selected lightweight variants from widely used LLM families showed substantial agreement with expert consensus when provided with guideline content, although findings reflect performance in a controlled, guideline-fed setting and require specialist validation before clinical application.