Bias Compensating Reinforcement Learning Control with Feedforward Adaptation for HVAC Systems

Around 76% of US electricity consumption is attributed to buildings [2]. Optimal control of the heating, ventilation, and air conditioning (HVAC) systems is essential for energy efficient buildings and thermal comfort of the occupants. However, modeling building HVAC systems for optimal control design purposes is challenging as such systems employ a variety of complex equipment. The presence of external disturbances arising from unknown heat gains, occupancy variations, light sources, and outside weather changes, further adds to the difficulty. Reinforcement learning (RL) has recently been popular in HVAC controls owing to its model-free capabilities. However, the treatment of disturbances is a harder problem in the RL framework because the presence of unknown disturbances can lead to inconsistencies in the associated learning equation that results in estimation bias [1]. Different from the game-theoretic approaches [4] and output regulation framework [5] employed in RL, we present a two-step design approach that consists of a bias compensating feedback controller and a feedforward adaptation mechanism both of which prevent estimation bias from incurring in the presence of unknown disturbances. The presented scheme is entirely output feedback driven and does not require the precise knowledge of system parameters (only the sign of the high-frequency gain is needed). Preliminary results have been obtained that show the effectiveness of the proposed scheme.