An Online Learning Algorithm for Demand Response in Smart Grid

Demand response program with real-time pricing can encourage electricity users toward scheduling their energy usage to off-peak hours. A user needs to schedule the energy usage of his appliances in an online manner since he may not know the energy prices and the demand of his appliances ahead of time. In this paper, we study the users' long-term load scheduling problem and model the changes of the price information and load demand as a Markov decision process, which enables us to capture the interactions among users as a partially observable stochastic game. To make the problem tractable, we approximate the users' optimal scheduling policy by the Markov perfect equilibrium (MPE) of a fully observable stochastic game with incomplete information. We develop an online load scheduling learning (LSL) algorithm based on the actor-critic method to determine the users' MPE policy. When compared with the benchmark of not performing demand response, simulation results show that the LSL algorithm can reduce the expected cost of users and the peak-to-average ratio in the aggregate load by 28% and 13%, respectively. When compared with the short-term scheduling policies, the users with the long-term policies can reduce their expected cost by 17%.