1884-P: Enabling Small Language Models to Count Carbohydrates Using Meal Descriptions from People with Type 1 Diabetes

Introduction and Objective: Carb estimating for meal insulin dosing is challenging and often error-prone in daily life with T1D. Large language models (LLMs) can be leveraged but accuracy is unknown. Small language models (SLMs) can run directly on mobile devices but perform worse than LLMs. We assessed carb counting accuracy of a LLM and 2 SLMs before and after fine-tuning. Methods: We used 6518 self-reported meal phrases with nutritionist carb estimates from the Type 1 Diabetes EXercise Initiative study from 337 participants and an LLM (GPT4.1) to generate 456260 chat-based queries asking for carb amounts across 7 different categories ranging from more to less detailed descriptions. We evaluated carb estimation accuracy using GPT4.1 and SLMs Qwen2.5-0.5B and SmolLM2-360M. SLMs were fine-tuned with Low Rank Adaptation (LoRA) on 5518 meals (386260 questions) using Chain-of-Thought reasoning. We evaluated accuracy on a subset of 7000 questions not used for fine-tuning. Results: GPT had median absolute error (AE) of 5 g [1, 13]. Fine-tuning reduced AE from 31 g [12, 73] to 7 g [1, 16] for Qwen (P<0.001) and 28 g [12, 51] to 7 g [1, 16] for SmolLM (P<0.001). From Tukey’s fences (k=3), all models yielded extreme outliers of AE>61 g - GPT 0.6%, Qwen 1.8% and SmolLM 2% of all estimates. Conclusion: Fine-tuning SLMs improves accuracy in carb estimations. Large errors in LLMs and SLMs mean they should be used with caution to guide carb estimations and meal insulin dosing. Disclosure S. Biswas: None. C. Mosquera-Lopez: Employee; Current; Insulet Corporation. L. Wilson: Research Support; Current; Eli Lilly and Company, Rezolute Bio, Dexcom, Inc. R.D. Tweet: Other - honorarium; Obesity and Weight Management Stigma Training via ADA Webinar; Ended; Lilly. P.G. Jacobs: Research Support; Current; Dexcom, Inc., Eli Lilly and Company. Advisory Panel; Current; Sequel Medtech. Consultant; Current; Roche Diabetes Care. Research Support; Current; SFC Fluidics. Consultant; Current; 4YouAndMe. Funding Leona M. and Harry B.