Enhanced Wolpertinger architecture for solving optimal control problems of reconfigurable battery systems

Reconfigurable battery systems (RBSs) offer flexibility in battery connections to address several critical issues in battery safety, balancing, degradation, energy delivery, etc. However, the optimal control of RBSs is complicated due to the nonlinear characteristics of batteries and the discrete-value states of switches. To solve RBS optimal control problems, while some traditional methods, such as rule-based methods, graph theory-based methods, dynamic programming, and heuristic algorithms, can be deployed, some assumptions are commonly needed to simplify the system modeling or problem formulation. Deep reinforcement learning (DRL) has shown the potential to solve RBS optimal control problems without such assumptions, but it still faces critical difficulties in dealing with large-scale discrete action spaces. To address this challenge, we propose to introduce and enhance the Wolpertinger architecture (WA) when using DRL for solving RBS optimal control problems. Specifically, three action embedding methods are designed to enable the training of WA. Then, the original WA is enhanced by deploying a higher-performance agent and a nearest neighbor search algorithm to improve the convergence optimality and computation efficiency, respectively. Case studies based on an experimentally verified RBS model demonstrate that the enhanced WA manages to effectively solve RBS optimal control problems and outperforms the original WA and other DRL methods in terms of the convergence speed and converged reward. Such advantages become increasingly pronounced as the system scales up. For even larger-scale practical battery systems (e.g., 50 cells), a multi-agent architecture is introduced to successfully solve the optimal control problem. Experimental tests on an embedded RBS control platform further confirm the superior real-time performance and potential applicability of the enhanced WA in practical RBS control scenarios.