Risk-Aware Model-Based Control

Figures

• 

  FIGURE 1. An illustration of the interaction within an RL agent and its environment.

• 

  FIGURE 2. An example to illustrate three risk metrics: Value at Risk (VaR), Conditional Value at Risk (CVaR), and the worst case. Intuitively, Var is the (1−α) quantile of the cost distribution, and CVaR is the expected value of the cost distribution’s upper (1−α) tail.

• 

  FIGURE 3. Block diagram of our model-based reinforcement learning method on run time. For each state st at time step t, we obtain a list of candidate actions sequence At from the CMA-ES optimiser. We choose the best actions sequence At⋆ from this list by calculating the CVaR value of each choice. To obtain this CVaR value, we use a Bayesian neural network (BNN) to predict a sufficient number of trajectories for modeling distribution of accumulated rewards. After the best actions sequence At⋆ is found, the first action in this actions sequence is truly performed by the agent (model predictive control, MDP). Training of the BNN dynamics model is ignored in this block diagram for simplicity. A more detailed description of the system can be seen in the pseudocode Algorithm 1.

• 

  FIGURE 4. Block diagram of our Eidos environment. Two randomly initialized deep neural networks represent the dynamics model and reward model of an environment, making it flexible to vary the complexity of the corresponding MDP.

• 

  FIGURE 5. Learning curves of the dynamics model based on different warm-up data (Average of three trials), where the metric of loss is the sum of squared errors (SSE) with L2 regularization. In most cases, model-free methods significantly improve the dynamics model learning.

• 

  FIGURE 6. Predicted trajectories with RAMCO, with all dimensions of a state averaged over to a scalar. After a random warm-up we predict states recursively with the dynamics model. Each prediction and predictions average are in light and dark colors, and the baseline in dots. Through this visualization of predictive trajectories, we claim that this dynamics model meets the requirements of our control framework and can support our system to produce risk-aware control.

• 

  FIGURE 7. Dynamics model multi-step prediction error, with different RL environments (introduced in the supplementary material) and training epochs (20, 60, 200). We calculate the validation loss in Eq. 7 for one step, five steps, 10 steps, 50 steps, and 100 steps of prediction length. The smaller the value is, the more accurate the prediction makes, and the lighter color is shown. All the results are averaged after three trials.

• 

  FIGURE 8. Accumulated rewards from each episode of AntX for different algorithms. RAMCO (SAC) and RAMCO (PPO) stand for using SAC and PPO as training data generators in our RAMCO control framework, respectively. These two model-free agents are trained from scratch. RAMCO (Random) means RAMCO with random data for training of the dynamics model. Although our method uses 2×104 steps to warm-up, it can outperform all other methods (except PETS) in its very first controlled iteration. The use of SAC as the warm-up policy can further enhance the quality of results when compared to PPO warm-ups. Our methods also show their advantage of low variance.

• 

  FIGURE 9. A histogram showing the distribution of final returns among different algorithms. Data of the warm-up phase of RAMCO and the corresponding period of other methods are dropped. We can see from the figure that not only the average of our returns is outstanding, but also the variance. While the random-based RAMCO has shown an exceptional average return, the PPO-based RAMCO shows the lowest variance among all the algorithms, which could be a very useful property in risk-sensitive applications. The SAC-based RAMCO has a variance between the two.

• 

  FIGURE 10. Training curves for Eidos with 10 state dimensionality. The PPO-version of RAMCO outperforms all other methods after 6×104 steps of warm-up. Among model-free algorithms, DDPG produces the closest result to ours but has a very unrobust training behavior and worse final return. Our RAMCO method (PPO-warm-up version) surpasses all other model-based methods including PETS and especially show overwhelming superiority to MBMF.

• 

  FIGURE 11. Training curves for Eidos with 100 state dimensionality. The PPO-version of RAMCO continuous to show its advantages over all other model-based and model-free methods. In this case, model-free methods begin to show its advantages on scalability while other model-based methods start to show their shortage. However, RAMCO based on PPO-generated training data still outperforms other state-of-the-art model-free methods including PPO, DDPG and SAC and model-based methods including PETS, MBMF and MBPO after the warm-up phase.

• 

  FIGURE 12. Training curves for Eidos with 1,000 state dimensionality. RAMCO (Random) shows performance similar to PETS, which are still better than the model-based method MBMF and hybrid method MBPO. The PPO-warm-up version of RAMCO reached its highest returns during the warm-up phase, and its performance degrades afterwards. Therefore, PPO-warm-up RAMCO excels while dealing with scalability with its flexible control framework in real-world applications where the back-end warm-up policy can be adopted directly.