Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems

Higher penetration of intermittent solar photovoltaic (PV) systems in the distribution grid results in frequent voltage fluctuations. The conventional voltage regulating devices operating on a slow-timescale need to be supplemented with the fast-operating smart inverters with adjustable reactive power setpoints. Complete and accurate information about distribution network topology and line parameters is necessary for conventional model-based Volt-VAR control (VVC) methods. However, such information is often unavailable. To tackle these challenges, a reinforcement learning-based two-timescale VVC algorithm is proposed in this paper that jointly controls the conventional voltage regulating devices at the slow-timescale and the smart inverters at the fast-timescale. Our proposed VVC algorithm simultaneously minimizes voltage violation costs and system operation costs in a model-free manner utilizing historical operational data. Two hierarchically organized agents are set up for the slow-timescale and fast-timescale problems, which are coupled through a communication scheme. The two sets of control policies are learned concurrently by a deep deterministic policy gradient and multi-agent soft actor-critic algorithm respectively. Comprehensive numerical studies performed with the IEEE 123-bus distribution test feeder show that the proposed framework can identify near optimal control actions of voltage regulating devices and smart inverters in real-time operations.