Reinforcement learning–enabled multi–service coordinated control of energy storage systems

Against the backdrop of rapidly increasing renewable energy penetration, amplified active-power fluctuations have markedly intensified the demand for both frequency regulation and peak-shaving services. As a pivotal flexibility resource, battery energy storage systems (BESS) are required to simultaneously satisfy stringent requirements for high tracking accuracy, fast dynamic response, and degradation-aware operation under multi-service stacking scenarios. Targeting the progressively refined spot market and ancillary service regulations in Shanxi Province, China, this paper proposes a three-layer “Rule–Intelligence–Safety” coordinated control framework for multi-service operation of energy storage. In the proposed architecture, the Rule Layer computes a revenue-oriented station-level target power setpoint based on primary and secondary frequency regulation mechanisms and outputs marginal degradation cost signals for each storage unit. The Intelligence Layer employs a deep reinforcement learning (DRL) framework—specifically, the Proximal Policy Optimization (PPO) algorithm—to learn an intra-station power allocation policy that achieves an adaptive trade-off between command-tracking accuracy and battery degradation cost. The Safety Layer enforces hard operational constraints through rapid feasibility checking and decision rectification, ensuring that all dispatched commands remain within safe operational envelopes. Training and testing on real historical frequency data from the North China power grid demonstrate that the proposed method reduces battery degradation cost by up to 13.11% compared to conventional proportional allocation strategies, while maintaining near-perfect frequency regulation accuracy (99.98%). The proposed framework offers a practical and scalable solution for enhancing the whole-life-cycle profitability of energy storage stations providing stacked ancillary services.