The AI Response to the 2025ACC/AHA/ACEP/NAEMSP/SCAI Guideline for the Management of Patients With Acute Coronary Syndromes: A Comparative Evaluation of Five Large Language Models

Background and Aim: Cardiovascular diseases, particularly chronic coronary syndromes (CCS), remain a leading cause of morbidity and mortality. Large language models (LLMs) have emerged as promising tools for processing medical information and supporting clinical decision-making. However, systematic comparative evaluation of their performance on current cardiology guidelines remains limited. This study aimed to comparatively evaluate the quality, guideline adherence, and clinical applicability of responses generated by five LLMs to clinical questions derived from the 2025 ACC/AHA/ACEP/NAEMSP/SCAI Guideline for the Management of Patients With Acute Coronary Syndromes. Methods This was a cross-sectional, observational, methodological simulation study. A pool of 20 open-ended clinical questions was developed based on 2025 ACC/AHA/ACEP /NAEMSP /SCAI Guideline for the Managementof Patients With Acute Coronary Syndromes, covering diagnostic and therapeutic domains, and validated by a panel of three independent cardiologists with over 15 years of clinical experience. Questions were submitted to five LLMs Gemini, DeepSeek, Claude, Copilot, and ChatGPT using identical Turkish-language prompts via publicly available web interfaces in April 2026. Model identifiers were removed to ensure evaluator blinding. Five independent expert cardiologists scored each response on five domains Comprehensiveness, Guideline Appropriateness, Understandability, Clinical Applicability, and Overall Scoreusing a 5-point Likert scale. Interrater reliability was assessed via intraclass correlation coefficients (ICC) and Cronbach's alpha. Group comparisons were performed using linear mixed-effects models with Bonferroni correction. Results Interrater reliability was acceptable to good across all domains (ICC range: 0.632–0.762; Cronbach's alpha > 0.80 for all domains). The LLM variable exerted a statistically significant effect across all five evaluation domains (all p < 0.001). DeepSeek achieved the highest estimated marginal means in all domains, including Overall Score (EMM = 4.87 ± 0.22), significantly outperforming all other models (p < 0.05 for all pairwise comparisons). Gemini demonstrated the lowest performance across all domains (Overall Score EMM = 2.75 ± 0.24). ChatGPT and Claude showed comparable performance in all domains (Overall Score: EMM = 3.79 ± 0.22 vs. 3.61 ± 0.23; p = 1.000). Diagnostic questions yielded significantly higher Understandability scores than treatment questions (EMM 3.736 vs. 3.512; mean difference = 0.224, 95% CI: 0.067–0.381; p = 0.005), while no significant model × question category interaction was observed in any domain. Conclusion Although the DeepSeek model demonstrated superior performance specifically within the context of the 2025 AHA Guidelines, this could be attributed to Chain of Thought reasoning. Nevertheless, the overall reliability of large language models in clinical applications remains variable, and future studies should focus not merely on information retrieval, but on their impact on clinical patient management.