Black Box, White Coat: Analyzing Artificial Intelligence’s Synthesis of Pancreatic Cancer Guidelines

Abstract Introduction This study evaluated the ability of common AI models to synthesize pancreatic adenocarcinoma (PDAC) management recommendations based on primary literature. Methods Sixteen questions regarding PDAC "Disease Workup," "Management by Staging," "Palliative Care," and "Longitudinal follow-up," were generated using ASO and NCCN guidelines. Each question was asked to ChatGPT 4o, Perplexity, Copilot, Gemini, and OpenEvidence, with instruction to synthesize recommendations based solely on primary literature. Responses were recorded verbatim, and their concordance to accepted guidelines was rated on a 6-point scale by two blinded surgical oncologists (0=incorrect, 1=insufficient/missing essential elements, 2=insufficient, missing minor elements, 3=concordant, 4=overly inclusive, added unnecessary elements, 5=over-conclusive with hallucinations). Concordance was defined as the percentage of "concordant" ratings out of the total number of responses. Results were analyzed using Cohen Kappa and one-way ANOVA, with inter-rater agreement defined as scores within 1 point. Results Inter-rater Cohen kappa was 0.73, indicating good agreement. The overall mean answer rating was 3.2. Individual model means were: ChatGPT 3.6, Perplexity 3.1, Gemini 3.2, Copilot 2.9, and OpenEvidence 3.1. OpenEvidence displayed the highest response concordance rate (87.5%, p=0.001), while ChatGPT 4o had the lowest (18.8%). Concordance rates for Perplexity, Copilot, and Gemini were 71.9%, 65.6%, and 56.3%, respectively. No models produced an "incorrect" answer. ChatGPT and Copilot produced hallucinations at 15.6% and 3.2% of responses, respectively. Conclusion Although certain models provide reliable information, AI can provide non-concordant recommendations when attempting to synthesize PDAC management recommendations. Further evaluation is necessary before safely integrating AI into clinical guideline development.