Large language models for complex decision-making in colorectal cancer with peritoneal metastases

Background Large language models (LLMs) are increasingly evaluated for oncology decision support, but their performance in colorectal cancer with peritoneal metastases (CRC-PM) remains uncertain. We benchmarked 10 contemporary LLMs against an expert-consensus framework. Methods Forty-eight synthetic vignettes were derived from a 2026 CRC-PM consensus guideline, including 36 primary scenarios and 12 counterfactual scenarios forming six pairs. Each model completed every vignette three times using a frozen zero-shot structured prompt. The primary endpoint was prompt-aligned structured Full concordance; contradiction-only and full-semantic adjudication were sensitivity analyses. Two blinded attending colorectal surgeons, each with 5 years of clinical experience, independently validated the reference classifications. Results Among 1,080 primary responses, structured scoring classified 609 (56.4%) as Full, 117 (10.8%) as Partial, and 354 (32.8%) as Discordant. Model-specific structured Full concordance ranged from 49.1% to 63.0%; neither the Friedman test ( P = .104) nor clustered GEE ( P = .186) supported a robust model effect. Contradiction-only scoring was similar (56.2% Full), whereas full-semantic adjudication reduced Full concordance to 45.5%. Excluding prompt-development scenarios lowered absolute concordance but did not change the inferential conclusion. Expert agreement was 93.8%–97.9% across decision domains (Cohen kappa 0.898–0.948). Counterfactual and metachronous risk-management analyses showed wide uncertainty, and 21 clinically consequential discordance flags arose from five scenarios. Conclusions LLM performance in CRC-PM was sensitive to scoring definition and did not establish a stable between-model hierarchy. Prospective clinician-plus-LLM studies are required before clinical benefit can be inferred.