A Human-Machine Reinforcement Learning Method for Cooperative Energy Management

The increasing penetration of distributed energy resources and a large volume of unprecedented data from smart metering infrastructure can help consumers transit to an active role in the smart grid. In this article, we propose a human-machine reinforcement learning (RL) framework in the smart grid context to formulate an energy management strategy for electric vehicles and thermostatically controlled loads aggregators. The proposed model-free method accelerates the decision-making speed by substituting the conventional optimization process, and it is more capable of coping with the diverse system environment via online learning. The human intervention is coordinated with machine learning to: 1) prevent the huge loss during the learning process; 2) realize emergency control; and 3) find preferable control policy. The performance of the proposed human-machine RL framework is verified in case studies. It can be concluded that our proposed method performs better than the conventional deep Q-learning and deep deterministic policy gradient in terms of convergence capability and preferable result exploration. Besides, the proposed method can better deal with emergent events, such as a sudden drop of photovoltaic (PV) output. Compared with the conventional model-based method, there are slight deviations between our method and the optimal solution, but the decision-making time is significantly reduced.