Machine-Learning Aided Optimal Customer Decisions for an Interactive Smart Grid

In this paper, a hierarchical smart grid architecture is presented. The concept of smart home is extended in two aspects: 1) from traditional households with smart devices, such as advanced metering infrastructure, to intelligent entities with instantaneous and distributive decision-making capabilities; and 2) from individual households to general customer units of possibly large scales. We then develop a hidden mode Markov decision process (HM-MDP) model for a customer real-time decision-making problem. This real-time decision-making framework can effectively be integrated with demand response schemes, which are prediction based and therefore inevitably lead to real-time power-load mismatches. With the Baum-Welch algorithm adopted to learn the nonstationary dynamics of the environment, we propose a value iteration (VI)-based exact solution algorithm for the HM-MDP problem. Unlike conventional VI, the concept of parsimonious sets is used to enable a finite representation of the optimal value function. Instead of iterating the value function in each time step, we iterate the representational parsimonious sets by using the incremental pruning algorithm. Although this exact algorithm leads to optimal policies giving maximum rewards for the smart homes, its complexity suffers from the curse of dimensionality. To obtain a low-complexity real-time algorithm that allows adaptively incorporating new observations as the environment changes, we resort to Q-learning-based approximate dynamic programming. Q-learning offers more flexibility in practice because it does not require specific starting and ending points of the scheduling period. Performance analysis of both exact and approximate algorithms, as compared with the other possible alternative decision-making strategies, is presented in simulation results.