Evaluating Reduced Order Models for Training Reinforcement Learning Agents for Building HVAC Control

Abstract Reinforcement learning (RL) for controlling building heating, ventilation, and air conditioning (HVAC) systems has shown promise in delivering energy-efficient, energy-flexible and resilient buildings. However, training RL agents in real-time in buildings is generally considered time and cost prohibitive. RL agents can be trained offline in simulated environments, reducing the time to performance once online, but complex models for simulation can be difficult to generate. What level of complexity is required in an offline training model to produce acceptable online results is unknown. This work seeks to determine how simplified models can be used to train RL agents for building HVAC control. The work uses the EnergyPlus simulation environment paired with HVAC systems modeled in Modelica to act as the real-world analogue environment. Simplified reduced-order models of the building were developed using the lumped capacitance method. The results show that an RL agent trained using Deep Q Learning on a simplified building model can be transferable to a more complex building simulation environment with comparable performance. In virtually all cases, the RL agents were able to deliver improved performance over the baseline proportional controller. Online learning gave mixed results in terms of enhancing the policy transfer process. The implementation in this paper was more suitable for improving the performance of poor policy transfer rather than fine-tuning already well-performing policies.