Application of deep Q-networks for model-free optimal control balancing between different HVAC systems

A deep Q-network (DQN) was applied for model-free optimal control balancing between different HVAC systems. The DQN was coupled to a reference office building: an EnergyPlus simulation model provided by the U.S. Department of Energy. The building was air-conditioned with four air-handling units (AHUs), two electric chillers, a cooling tower, and two pumps. EnergyPlus simulation results for eleven days (July 1–11) and three subsequent days (July 12–14) were used to improve the DQN policy and test the optimal control. The optimization goal was to minimize the building’s energy use while maintaining the indoor CO2 concentration below 1,000 ppm. It was revealed that the DQN—a reinforcement learning method—can improve its control policy based on prior actions, states, and rewards. The DQN lowered the total energy usage by 15.7% in comparison with the baseline operation while maintaining the indoor CO2 concentration below 1,000 ppm. Compared to model predictive control, the DQN does not require a simulation model, or a predetermined prediction horizon, thus delivering model-free optimal control. Furthermore, it was demonstrated that the DQN can find balanced control actions between different energy consumers in the building, such as chillers, pumps, and AHUs.