Evaluating the utility of large language models for diabetes education: a specialist-reviewed cross-sectional study

Introduction More than 500 million adults live with diabetes, and its daily management falls largely to patients themselves. Many now bring those questions to artificial intelligence (AI) chatbots, which answer instantly and confidently, but not always correctly. That matters more where specialist advice is hard to obtain. We evaluated ChatGPT-3.5 in diabetes clinics in Karachi, Pakistan, examining what patients ask and how specialists judge the answers. Methods 51 adults with type 2 diabetes at 3 clinics asked 137 unscripted questions; 45 (88.2%) required real-time interpretation. 3 endocrinologists assessed 126 metabolic responses and 3 ophthalmologists assessed 11 ocular responses for clarity and completeness (1–5), and correctness, safety and up-to-dateness (yes/no). Agreement used intraclass correlation coefficient (ICC)(2,k), Fleiss’ κ and prevalence-robust Gwet’s agreement coefficient (AC). Results By majority consensus, metabolic responses were correct (98.4%), safe (99.2%) and up-to-date (100%); mean clarity 4.66 and completeness 4.53. Despite 91.0–95.2% observed agreement and Gwet’s AC of 0.74–0.95, chance-corrected coefficients were low (ICC 0.13 and 0.25; κ −0.05 to 0.08), indicating ceiling effects and imprecise mean ratings rather than absent concordance. The exploratory ocular subset performed worse: 6/11 responses correct, 9/11 safe, 7/11 current; both responses meeting the prespecified hallucination criterion occurred here. 50 of 51 participants rated responses informative, detailed and empathic. Conclusion ChatGPT-3.5 may complement supervised diabetes education, but reliability appeared to vary by clinical domain and high mean ratings obscured the spread in specialist judgements. Participants endorsed answers they could not independently verify. Evaluations of patient-facing chatbots should validate each clinical domain separately before deployment. Trial registration number NCT05883072.