Hierarchical Reinforcement Learning-Based Energy Management for Hybrid Electric Vehicles with Gear-Shifting Strategy

The energy management strategy (EMS) is a core technology for improving the fuel economy of hybrid electric vehicles (HEVs). However, the coexistence of both discrete and continuous control variables, along with complex physical constraints in HEV powertrains, presents significant challenges for the design of efficient EMSs based on deep reinforcement learning (DRL). To further enhance fuel efficiency and coordinated powertrain control under complex driving conditions, this study proposes a hierarchical DRL-based EMS. The proposed strategy adopts a layered control architecture: the upper layer utilizes the soft actor–critic (SAC) algorithm for continuous control of engine torque, while the lower layer employs a deep Q-network (DQN) for discrete gear selection optimization. Through offline training and online simulation, experimental results demonstrate that the proposed strategy achieves fuel economy performance comparable to dynamic programming (DP), with only a 3.06% difference in fuel consumption. Moreover, it significantly improves computational efficiency, thereby enhancing the feasibility of real-time deployment. This study validates the optimization potential and real-time applicability of hierarchical reinforcement learning for hybrid control in HEV energy management. Furthermore, its adaptability is demonstrated through sustained and stable performance under long-duration, complex urban bus driving conditions.