Not all artificial intelligence large language models are created equal: heterogeneous performance in therapeutic decision-making for recurrent epithelial ovarian cancer

Objective To evaluate the concordance between therapeutic recommendations generated by three widely used consumer large language models (Gemini, Microsoft Copilot and ChatGPT) and the decisions of a Multidisciplinary Tumor Board in the management of first recurrence of high-grade serous ovarian cancer. Secondary objectives were to characterise recurrent error patterns and to assess the safety of model-generated recommendations using a three-tier taxonomy. Methods Retrospective observational analysis of 44 consecutive patients with first recurrence of high-grade serous ovarian cancer treated at a tertiary referral centre. For each case, an identical, anonymised, structured zero-shot prompt was submitted to the three models. The Multidisciplinary Tumor Board decision was the reference comparator. Overall concordance and concordance for the chemotherapy backbone, maintenance/targeted-agent class and surgical indication were compared between models using Cochran's Q and pairwise McNemar tests (Holm-adjusted). Discordant recommendations were categorised as true hallucinations, unsafe, or suboptimal but guideline-concordant. Results Concordance varied substantially across models (Cochran's Q p<0.001): Gemini 77.3% (95% CI 62.2–88.5), Microsoft Copilot 36.4% (22.4–52.2), ChatGPT 18.2% (8.2–32.7); Gemini differed significantly from both other models (Holm-adjusted p<0.001), whereas the difference between Microsoft Copilot and ChatGPT did not reach significance after adjustment (p=0.057). Component-level analysis localised the divergence to systemic therapy: surgical-indication agreement was uniformly high (93.2–95.5%; p=0.85). All three models generated unsafe recommendations of PARP inhibitor re-use in patients with documented prior progression on PARP inhibitor maintenance with marked differences in frequency (Microsoft Copilot 10, ChatGPT 3, Gemini 2; 15 of 74 total discordances). The remaining discordances were suboptimal but guideline-concordant. Conclusions Consumer large language models showed heterogeneous and clinically meaningful differences in reproducing Multidisciplinary Tumor Board decisions in recurrent ovarian cancer; none was free of unsafe recommendations. Their current performance does not support independent clinical use, and expert supervision is required to ensure patient safety.