Joint Optimization of Discrete and Continuous Reservoir Control Strategies Using Recent Experience-Based Hybrid Maximum Entropy Deep Reinforcement Learning

Summary Petroleum resources remain the cornerstone of global energy supply, and the joint optimization of discrete and continuous measures is crucial to reservoir development. The coordination of discrete layer-blocking decisions and continuous well-control operations poses significant challenges, where existing methods often convert discrete variables into continuous ones and require reoptimization for different reservoir development scenarios, hindering efficient and accurate optimization. In this paper, we propose a hybrid maximum entropy deep reinforcement learning (DRL) optimization method guided by recent experience (GRE-HMEDRL), which is used for the joint optimization of discrete layer selection and continuous well control parameters. The joint optimization problem is modeled as a Markov decision process (MDP) with a hybrid action space of discrete and continuous actions, integrating discrete and continuous soft actor-critic (SAC) algorithms within a maximum entropy framework. The agent interacts with the reservoir numerical simulator in real time, accumulating historical experiences, and the recent-experience sampling mechanism dynamically adjusts the sampling range, enabling rapid training of a hybrid action policy that maximizes the net present value (NPV) of reservoir development. The hybrid action policy extracts latent state representations of the reservoir through a deep convolutional neural network (CNN) shared by both discrete and continuous action policies, outputting layer blocking and well control variables for joint optimization. Extensive experiments on two 3D reservoir models demonstrate that GRE-HMEDRL consistently outperforms gradient-based, evolutionary algorithms, and existing DRL methods in terms of optimization performance, achieving the highest NPV with greater oil recovery and reduced water production. Compared with baseline methods, GRE-HMEDRL exhibits faster convergence, improved robustness, and higher efficiency. Notably, the trained hybrid action policy demonstrates strong robustness and adaptability in offline deployment across diverse scenarios, including increased regulation frequency and well failures. Without retraining, it maintains near-optimal performance (within a 0.1–4.2% NPV gap) while avoiding the need for thousands of additional reservoir simulations, significantly reducing computational cost and enabling real-time application in practical reservoir management. In this paper, we use well control and layer selection as a representative example, but the GRE-HMEDRL framework is readily applicable to other coupled discrete-continuous optimization problems.