Hybrid Reinforcement Learning Optimization and Control of Steam-Assisted-Gravity-Drainage Injection-Production Operations under Subcool Constraints

Summary Subcool is a key indicator for evaluating steam-assisted-gravity-drainage (SAGD) operating performance and steam-chamber development. Effective subcool control helps maintain stable injection and production, reduces the risk of steam breakthrough, and supports more favorable steam-chamber development. Advances in downhole pressure and temperature monitoring make subcool information more readily available, providing a technical basis for feedback-based subcool regulation and proactive SAGD production control. However, existing comprehensive SAGD optimization methods usually focus on improving overall injection and production performance across multiple indicators. They are often designed for stage-based performance evaluation and involve long feedback periods, slow control updates, and delayed decisions. As a result, they have limited capability for feedback-based regulation of short-time-scale subcool fluctuations. Some local feedback control studies have improved the responsiveness of subcool regulation, while the coordinated optimization of long-term development indicators, such as steam injection intensity, production performance, steam-chamber development, and economic return, is less directly incorporated. To meet the requirements of stage-level development optimization and immediate subcool feedback, we propose a hybrid twin delayed deep deterministic policy gradient (TD3) and proportional-integral-derivative (PID) optimization and control framework to coordinate long-term development performance improvement with short-period subcool regulation. In this framework, the TD3 agent learns injection and production operating patterns from policy interaction trajectories. Guided by long-term development benefits and stage-based production performance, TD3 recommends stage-level steam injection rates and baseline electric submersible pump (ESP) frequencies. The PID controller uses dynamic subcool feedback to correct ESP frequency over a shorter time scale, thereby adjusting production intensity and improving the response to subcool deviations. For a 5-year optimized production period, TD3-PID maintains subcool more stably within the target range and forms a more uniform and laterally extended steam chamber compared with the particle swarm optimization (PSO) benchmark. The final mean net present value (NPV) increases by 5.57%, oil recovery increases by 3.59%, and cumulative steam/oil ratio (cSOR) decreases by 10.32%. In an extended inflow-control-valve (ICV) scenario for reservoir heterogeneity, the framework also shows potential for more complex injection and production control involving localized steam redistribution. The proposed TD3-PID framework integrates stage-level injection and production parameter optimization, long-term development performance improvement, and short-term subcool deviation correction into the same closed-loop decision process. It addresses the slow response of conventional comprehensive optimization methods to operating-state fluctuations and the limited ability of single feedback control to optimize long-term development performance. This extends SAGD control from stage-based parameter optimization or local feedback regulation toward dynamic closed-loop control across multiple indicators and control levels. The results provide a methodological basis for developing more automated SAGD injection optimization, pump-frequency adjustment, and production control strategies.