2863-LB: Evaluation of Large Language Models (LLMs) Pharmacotherapy Recommendations in Complex Type 2 Diabetes Management

Introduction and Objective: This project aimed to evaluate LLMs' recommendations for pharmacotherapy in the management of type 2 diabetes (T2D) in alignment with the 2026 American Diabetes Association’s (ADA) Standards of Care. The findings provide insights into the feasibility and reliability of integrating generative artificial intelligence with T2D and cardio-kidney-metabolic care. Methods: Four LLMs were used: ChatGPT 5.2, Gemini 3 Pro, Copilot 365, and Claude Opus 4.6. Each LLM generated five rounds of outputs using a published case of T2D with CKD. Three clinical pharmacists with CDCES used a 37-question binary rubric to evaluate LLM responses. Logistic regression was used to estimate high concordance (Overall Score ≥ 80%, Safety Domain ≥ 80%, and no critical errors). Percentage scores were compared across LLMs using the Kruskal-Wallis test. When statistically significant, Bonferroni-corrected pairwise comparisons were performed. Results: There is variability in multiple measures between LLM models, but there was initial concordance with ADA standards. Conclusion: Significant inter-model variability was observed in Safety, Dosing, and Overall performance. These findings signal domain-specific differences of LLMs in recommending diabetes pharmacotherapy. Further investigation is needed to critically evaluate LLM recommendations in broader T2D care and to more precisely delineate inter-model differences. Disclosure M.Donoghue: None. K.O’brien: Advisory Panel; Ended; Sanofi, Lilly Diabetes, Insulet Corporation, Speaker's Bureau; Current; Insulet Corporation, CeQur SA. D.Isaacs: Consultant; Current; Medtronic, Tandem Diabetes Care, Inc., Speaker's Bureau; Current; Dexcom, Inc., Abbott, Insulet Corporation, Ascensia Diabetes Care, CeQur SA, MannKind Corporation, Novo Nordisk, Lilly. J.Clements: n/a. C.Young: Advisory Panel; Ended; Sanofi.