Key points are not available for this paper at this time.
Optimally controlling energy storage systems is essential for the integration of renewables and electromobility in future power grids. Such control schemes may need to satisfy lookahead requirements for the batteries state-of-charge so as to ensure a smooth grid operation. Model predictive control (MPC) is traditionally used to solve these type of problems suffering however from long runtimes and the need of fine-tuned forecasts. Reinforcement Learning (RL) approaches are considered as alternatives but usually cannot guarantee constraints satisfaction. To overcome the limitations of both MPC and RL, in this work, a novel method that integrates RL into MPC is proposed leading to a fast real-time control scheme that aims to enforce near-future constraints satisfaction and accounts for lookahead constraints in the current decisions. Due to the inclusion of the RL-based learned value function in the MPC cost, traditional optimization solutions do not apply and the application of the cross-entropy-method (CEM) is suggested. Its runtime is reduced to the time scale of seconds by employing appropriate initialization based on RL decisions as well as an exponentially decaying amount of samples. Numerical comparisons show that the proposed method outperforms MPC and other safe RL schemes in terms of cost performance and constraints violations.
Locher et al. (Mon,) studied this question.