Personalized decision making for coronary artery disease treatment using offline reinforcement learning

Figures

• 

  Fig. 1. Distribution of number of transitions and reward values per episode. Logarithmic distribution of (a) number of transitions per episode and (b) accumulated rewards per episode in training and test sets.

• 

  Fig. 2. Expected rewards of different policies using the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\pi }_{{B}_{best}}$$\end{document}πBbest as behavior policy. This figure illustrates the expected rewards from optimal QL-based policies and physician policies as the number of states (clusters) increases. Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\pi }_{{B}_{best}}$$\end{document}πBbest (the policy best representing physicians’ decision-making patterns) is used as the behavior policy for off-policy evaluation. Blue circles and green triangles represent the stochastic optimal and physician policies, respectively, while orange squares and red rhombuses represent the greedy optimal and physician policies. Panel (a) shows results from the training dataset, and panel (b) shows results from the test dataset.

• 

  Fig. 3. Expected rewards of different policies using the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\pi }_{{B}_{n}}$$\end{document}πBn as behavior policy. This figure illustrates the expected rewards from optimal QL-based policies and physician policies as the number of states (clusters) increases. Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\pi }_{{B}_{n}}$$\end{document}πBn (the physician policy with the same number of clusters as n) is used as the behavior policy for off-policy evaluation. Blue circles and green triangles represent the stochastic optimal and physician policies, respectively, while orange squares and red rhombuses represent the greedy optimal and physician policies. Panel (a) shows results from the training dataset, and panel (b) shows results from the test dataset.

• 

  Fig. 4. Heatmap of action probabilities in the optimal and physician policies (number of states = 84) and a simple average policy of both. This figure presents a heatmap showing the probability of each possible action across different states for three policies: the Optimal Policy derived from Traditional Q-Learning, the Physician Policy based on the physicians’ decision-making model, and an Average Policy combining the two. All policies are evaluated in a setting with 84 states.

• 

  Fig. 5. Feature importance for each cluster (heatmap color) calculated using the Shapley method. This heatmap shows the relative importance of the top 20 features calculated using the Shapley method. The feature center for each state (the center of the cluster in the k-means results) is shown inside the corresponding cell. The chosen optimal treatment for each state is color-coded on the state number (Red: CABG, Blue: PCI, Green: MT). The trained XGBoost model details for calculating Shapley values are in Supplementary Table 4.

• 

  Fig. 6. t-SNE visualization of different greedy policies. These plots use t-SNE visualization to illustrate the treatment decisions made by each greedy policy in the test set. Green, blue, and red dots represent Medical Therapy alone, PCI, and CABG, respectively. Each panel shows decisions from the following: (a) Real treatments received, (b) the best behavior policy from physicians’ decisions, (c) the best Traditional QL policy, (d) DQN policy, (e) CQL policy with α = 0.001, (f) CQL policy with α = 0.01, (g) CQL policy with α = 0.1, and (h) CQL policy with α = 0.5.

• 

  Fig. 7. Expected rewards gained by each greedy policy for different groups of patients stratified by sex, age, ACS, and APPROACH Jeopardy score. This figure stratifies the data into two distinct groups based on (a) Sex, (b) Age, (c) Acute Coronary Syndrome, and (d) APPROACH Jeopardy Score, comparing the expected rewards within each group across different policies.

• 

  Fig. 8. The schema of formulating the problem as an MDP. This figure illustrates the schema of formulating CAD treatment problem as an MDP. It shows possible transitions within each episode: a coronary angiography and its associated treatment can lead to (a) another coronary angiography (continuing the episode), (b) death from any cause (episode termination), or (c) neither of the above within the 3-year follow-up period (episode termination). Some components of this figure are AI-generated.

• 

  Fig. 9. Actions and rewards structure and formulation in each transition. This figure depicts the action space and rewards structure and formulation in each transition of the MDP model. Terminal states refer to states where the episodes end. Patients get a positive reward between 0 and 1 in the terminal state for the portion they lived (i.e., all-cause mortality reduces this positive reward). This ensures that patients receive positive rewards for the time lived following treatments. Negative rewards from MACE apply to all states, including terminal states (e.g., CV death would further reduce the overall reward). Some components of this figure are AI-generated.

• 

  Fig. 10. Overall methodology in training and evaluating RL models to optimize CAD treatment. This figure summarizes our methodology. Patient data, physician decisions, and subsequent outcomes were collected from various data sources. Using these sources, we structured the data as an MDP and optimized it using traditional Q-Learning, Deep Q-Network, and Conservative Q-Learning algorithms. The optimal policy was then compared to the physician decision model using off-policy evaluation. Some components of this figure are AI-generated.