Multicenter benchmarking of large language models for bladder cancer management recommendations from mpMRI reports

Large language models (LLMs) can interpret radiology reports, but their ability to generate guideline-concordant recommendations from bladder cancer multiparametric MRI (mpMRI) reports remains uncertain. Following TRIPOD-LLM, we retrospectively evaluated nine API-accessed LLMs using 590 mpMRI reports from three hospitals under two input conditions: original Chinese reports and clinician-translated, quality-controlled English reports. Two physicians independently rated each model-generated recommendation on a 1–5 ordinal scale. We analyzed 21,240 ratings using patient-level random-intercept cumulative link mixed models, with paired Wilcoxon tests as secondary exploratory analyses. The association between input condition and physician rating differed across LLMs (likelihood-ratio chi-square = 683.05, df = 8, P < 0.001). Compared with original Chinese input, clinician-translated English input was associated with higher ratings for chatgpt-o3-pro (OR 2.79, 95% CI 2.29–3.41), deepseek-v4-flash (OR 1.46, 95% CI 1.19–1.81), claude-opus-4-7 (OR 3.07, 95% CI 2.47–3.82), and gpt-5.5 (OR 1.88, 95% CI 1.53–2.32), but with lower ratings for claude-sonnet-4-5 (OR 0.29, 95% CI 0.23–0.36; all Holm-adjusted P < 0.001). Four LLMs showed no clear input-condition difference. Center-specific contrasts also depended on input condition and were materially unchanged after adjustment for age, sex, pathological T stage, tumor grade, tumor number, and maximum lesion diameter. Inter-rater agreement was substantial (quadratic weighted kappa range, 0.621–0.743). Readability did not track physician-rated recommendation quality. These findings argue against clinician-translated English as a universal optimization strategy and support model-specific validation of input conditions before clinical deployment. LLM-generated recommendations from mpMRI reports are best positioned as clinician-supervised, imaging-based decision support rather than standalone treatment advice.