Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial

Figures

• 

  Fig. 1. Schematic illustration of the AI system from development to deployment for dynamic insulin dosage titration for patients with T2D. a, Model development of the AI system—a model-based RL-DITR system consisting of ‘patient model’ and ‘policy model’. Left, we constructed a large multi-center EHR dataset consisting of records of long-term continuous clinical observation and medication of hospitalized patients with T2D. Middle, with the standardized time-series data as input, the patient model generated hidden state transition, status prediction and reward estimation. Right, the policy model is optimized by interacting with the patient model as an environment. b, Comprehensive evaluation of the AI system step-by-step for integration into the real-world clinical workflow. Left, we conducted multi-center retrospective studies, including quantitative and qualitative evaluations in the internal and external cohorts. Middle, a prospective study with test–retest was conducted in an academic hospital after AI deployment in the HIS. Right, a proof-of-concept feasibility trial was conducted to evaluate the glycemic control of and physician satisfaction with the AI system.

• 

  Fig. 2. Performance of AI model system in the prediction of patient state trajectory. a,b, Comparison of actual patient trajectories and model-based state roll-outs for patients from the internal test set (a) and the external test set (b). Each predicted value, based on an individual patient, is generated within K steps from the last timestep of the previous day (K = 7 for 1 d ahead of time). The blue curve is measured patient glucose values, and the orange curve is predicted glucose values. c,d, Correlation analysis of the predicted glucose value versus the actual glucose value generated using the AI glucose model in the internal test set (c) and the external test set (d). e,f, ROC curves showing performance of daily WTR prediction on the internal test set (e) (n = 20,961 treatment days) and the external test set (f) (n = 16,077 treatment days). Each predicted value is based on the last timestep of the previous day. Box plots show the median (center lines), interquartile range (hinges) and 1.5× interquartile range (whiskers) (bootstrapping with n = 1,000 resamples). Each value generated by our RL-DITR system represents an individual-level prediction. These were then aggregated to…

• 

  Fig. 3. Performance of AI treatment model in the insulin dosage prediction. a,b, Performance of daily treatment dosage prediction on the internal test set (a) (n = 42,037 insulin data points) and the external test set (b) (n = 32,484 insulin data points). Each predicted value was subsequently unrolled recurrently for K steps from the last timestep of the previous day (K = 7 for 1 d ahead of time). The error bars represent the 95% CIs. We aggregated the individual-level predictions to obtain population-level results. c,d, Comparison of actual treatment regimens and model-based treatment roll-outs of two individual patients from the internal test set (c) and the external test set (d). The blue curve is measured patient glucose values, and the orange curve is predicted glucose values given by the AI model. e,f, Association analysis of the patient outcome (for example, WTR) versus the dosage difference in treatment actions between the AI policy and the clinician policy for the internal test set (e) and the external test set (f). The dose excess, referring to the difference between the given and the AI model, suggested dose summed over per day for all patients. The shaded area represents the…

• 

  Fig. 4. Performance evaluation between the AI model and human physicians in retrospective studies. a–e, Comparison with quantitative metrics (a and b) and qualitative clinical evaluations (c–e) on insulin titration regimens given by the AI model and human physician groups in the internal cohort (n = 40 patients with T2D). The AI model was compared with three physician groups with different levels of clinical experience: group 1, junior physicians (n = 5); group 2, intermediate physicians (n = 5); and group 3, senior physicians (n = 5). a,b, Quantitative evaluation of dosage titration by expert panel consensus as reference (n = 226 insulin data points). a, Predicted error (MAE) of AI model and human physicians; b, dosage adjustment agreement of the AI model and human physicians evaluated separately by identical agreement (same direction, same dosage) and clinical agreement (same direction, dose difference ≤20%). c–e, Qualitative clinical performances were evaluated by the expert panel (n = 40 regimens) separately in effectiveness (c), safety (d) and overall acceptability (e). f–h, Performance comparison of the AI-generated and previously delivered insulin regimens in the test–retest…

• 

  Fig. 5. Performance evaluation of the AI model in the prospective study. a–d, The performance of effectiveness (a), safety (b), overall acceptability (c) and adoption rate of AI-generated regimen (d) evaluated by endocrinologists at the bedside at test–retest review in the prospective study (n = 40 regimens). The score scale of effectiveness and safety is 1–5. The adoption rate refers to the percentage of the AI regimens adopted by endocrinologists at the bedside for patient treatment. Bar graphs depict the mean ± s.e.m.

• 

  Fig. 6. A proof-of-concept feasibility trial to evaluate the AI system on glycemic control in patients with T2D. a, The baseline clinical characteristics of patients with T2D included in the proof-of-concept feasibility trial (n = 16). b, The capillary blood glucose of a patient with T2D during the treatment period. (I) Illustration of the seven-point glucose profile during the first and last 24 h of the treatment period. Statistical significance was determined by two-sided paired t-test: pre-breakfast ***P < 0.001; post-breakfast, pre-lunch, post-lunch, post-dinner and pre-bedtime **P < 0.05; pre-dinner *P < 0.10. (II) Mean daily capillary blood glucose. (III) Mean preprandial capillary blood glucose. (IV) Mean postprandial capillary blood glucose during the treatment period. The preprandial blood glucose target was 5.6–7.8 mmol L−1; the postprandial capillary blood glucose target was <10.0 mmol L−1. (II–IV) Line, median; error bar, interquartile; n = 16 patients. c, Average percentage of continuous glucose monitoring data within glycemic ranges throughout the treatment period. The percentage of continuous glucose measurement <3.0 mmol L−1, 3.0–3.8 mmol L−1, 3.9–10.0 mmol L−1, 10.1–13.9…

• 

  Extended Data Fig. 1. The development of the AI system. a, The sequential decision process with patient model and policy model in AI system. Given a trajectory, for the initial step, the representation function fR receives as input the past observations O1:t from the trajectory. The model is subsequently unrolled recurrently for K steps. At each step κ∈[1,K], the policy model π receives the hidden state \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${s}_{t+k-1}$$\end{document}st+k−1 and generates an action \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${a}_{t+k-1}$$\end{document}at+k−1. The dynamics function fT of patient model subsequently receives as input the hidden state \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs}…

• 

  Extended Data Fig. 2. The performance evaluation of the AI system for patient trajectory prediction using WTR for overall glucose variability. a, Visualization of the patient hidden states. Projection of patients’ hidden state embeddings onto PC 1 and PC 2, derived from principal component analysis (PCA). Each node indicates a patient state. The state distribution showed association with diabetic outcome, colored by glucose level distribution. The samples are 1000 patients from internal test dataset. PC: principal component. b, Illustration of reward function. It is a measurement of overall glucose variability that focus on the relationship between glucose variability and risks for hypo- and hyperglycemia. c and d, Performance of the AI model on assessment of WTR shown as AUC curves. c, internal test set and d, external test set. ROC curves showing the pre-prandial time, the postprandial and overall performance. e-f, Correlation analysis of the ratio of glycemia within target range (WTR) vs the estimated cumulative reward of the clinicians’ treatment actions. e, internal test set, and f, external test set. The shaded area represents the 95% confidence interval.

• 

  Extended Data Fig. 3. Performance of the AI for daily treatment dosage prediction and off-policy evaluation. a-f, Each column represents the performance of AI grouped by insulin types, including (a and b) short/rapid acting, (c and d) biphasic or premixed, and (e and f) long acting. Each predicted data is based on is generated within K steps based on the last time step of previous day (K = 7 for one day ahead of time). The bars represent the mean with 95% confidence intervals of MAE on the internal test set (n = 20,961 treatment days) and the external test set (n = 16,077 treatment days). MAE, mean absolute error; R2, coefficient of determination; PCC, Pearson’s correlation coefficient. a, c and e: the internal test set; b, d and f: external test set. g, Off-policy evaluation of RL-based model versus other SL-based and clinician methods in the internal test set in the AI development phase, measured by weighted importance sampling (WIS) score with standard deviation. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt}…

• 

  Extended Data Fig. 4. Performance evaluation of the AI system in the retrospective phase study of internal and external cohorts. a, Study design of internal cohort: 40 eligible T2D patients were included in the study. 80 treatment cases from 40 patients (2 per person) were randomly selected to compare the performance of quantitative metrics between the AI system and human physicians. 1 case for 40 patients was further selected for qualitative clinical evaluations (effectiveness, safety, and overall acceptability). b, Study design of external cohort: 45 T2D patients were collected, and a total of 796 insulin points were included in the external validation analysis. An assessment with quantitative metrics was conducted to compare the performance between treating physicians and AI by expert panel. 40 cases randomly selected from the total 338 cases were used for further qualitative evaluation (effectiveness, safety, and overall acceptability). After 2 weeks, a retest review was conducted. c, Demographics and baseline measurements of the patients in the internal (n = 40) and external (n = 45) cohorts. BMI, body mass index; A1c, glycated hemoglobin. Numerical variables were reported as mean±SD. d.…

• 

  Extended Data Fig. 5. Performance evaluation of AI system in the prospective deployment study. a, The user interface of AI deployment. b, Demographics and baseline measurements of patients in the deployment phase study (n = 20). BMI, body mass index; A1c, glycated hemoglobin. Numerical variables were reported as mean±SD.

• 

  Extended Data Fig. 6. Flow diagram of the proof-of-concept feasibility trial.