BackgroundLarge language models (LLMs) have demonstrated strong performance on general medical and surgical examinations; however, their capacity to accurately interpret and apply subspecialty clinical practice guidelines remains incompletely characterized.ObjectiveTo evaluate and compare the accuracy and consistency of two contemporary LLMs-Google Gemini and OpenEvidence-using multiple-choice questions (MCQs) derived directly from the 2022 American Society of Colon and Rectal Surgeons (ASCRS) Clinical Practice Guidelines for anorectal abscess, fistula-in-ano, and rectovaginal fistula.MethodsThirty guideline-based MCQs were developed and independently validated by surgeon reviewers. Each question was presented to both models under identical conditions without additional prompting. Accuracy was calculated with 95% confidence intervals and compared against chance performance (p0 = .25). Inter-model agreement was assessed using Cohen's kappa coefficient.ResultsBoth Gemini and OpenEvidence correctly answered 29 of 30 questions (96.7%; 95% CI, 0.83-0.999), significantly exceeding chance performance (P < .0001 for both models). Both models missed the same question, yielding perfect inter-model agreement (κ = 1.0).ConclusionIn this focused pilot study restricted to ASCRS anorectal disease guidelines, both LLMs demonstrated near-perfect and statistically equivalent accuracy. These findings suggest that contemporary LLMs can accurately apply subspecialty surgical guidelines within a narrow domain, though broader, multi-guideline evaluations are required before generalization.
Pandya et al. (Sun,) studied this question.