Towards Pareto-optimal energy management in integrated energy systems: A multi-agent and multi-objective deep reinforcement learning approach

Deep Reinforcement Learning (DRL) is effective in solving complex, non-linear optimization problems, which is particularly relevant in energy management within Integrated Energy Systems (IESs). However, DRL approaches conventionally focus on single-objective policy learning, which is inadequate for the multi-objective optimization tasks commonly encountered in IESs energy management. To improve this, these approaches typically combine multi-objectives, such as operating cost objective and safety objective into a single reward function using scalarization techniques. This reduces the fidelity and interpretability of the objective space and limits its applicability to a wide range of IESs energy management. To address these challenges, this paper presents a novel framework called Multi-Agent and Multi-Objective DRL (MAMODRL). This framework combines value function decomposition and policy gradient methods to achieve a Pareto-optimal solution. The IESs energy management is initially formulated as a multi-objective Markov decision process. Then, an advanced MAMODRL architecture is developed, which includes objective value function networks to facilitate policy optimization. Finally, based on the definition of dominance, Pareto frontier is approximated of IESs energy management. A case study suggests that the proposed approach is effective in solving the Pareto frontier for IESs energy management. To ensure the safe operation of the system, safety threshold is set at the Pareto frontier forming a Pareto optimization with safety conditions. Compared to traditional DRL approaches, the proposed approach is more flexible, interpretable, and capable of making multi-dimensional decisions.