Evaluation of Reinforcement Learning Control for Thermal Energy Storage Systems

This paper describes a simulation-based investigation of machine-learning control for the supervisory control of building energy systems. Model-free reinforcement learning control is investigated for the operation of electrically driven cool thermal energy storage systems in commercial buildings. The reinforcement learning controller learns to charge and discharge a thermal storage tank based on the feedback it receives from past control actions. The learning agent interacts with its environment by commanding the thermal energy storage system and extracts cues about the environment solely based on the reinforcement feedback it receives, which in this study is the monetary cost of each control action. No prediction or system model is required. Over time and by exploring the environment, the reinforcement learning controller establishes a statistical summary of plant operation, which is continuously updated as operation continues. The controller learns to account for the time-dependent cost of electricity (both time-of-use and real-time pricing), the availability of thermal storage, part-load performance of the central chilled water plant, and weather conditions. Though reinforcement learning control proved sensitive to the selection of state variables, level of discretization, and learning rate, it effectively learns a difficult task of controlling thermal energy storage and displays good performance. The cost savings compare favorably with conventional cool storage control strategies but do not reach the level of predictive optimal control.