Deployment-time action correction for steady-state reinforcement learning-based optimal power flow

This paper addresses the problem of constraint violations in reinforcement learning (RL) for power-system operation, with application to steady-state optimal-power-flow tasks. While RL-based controllers can achieve competitive reward performance, they may produce actions that violate operational constraints, limiting their practical applicability in steady-state optimal-power-flow (OPF) tasks. We investigate a deployment-time safety framework in which a continuous constraint-violation surrogate is learned directly from state–action–violation samples generated by the power-system simulator. Once the nominal Soft Actor-Critic (SAC) policy has been trained, it remains fixed, and candidate actions are corrected by minimizing their deviation from the original policy output while penalizing the violation predicted by the learned surrogate. The correction is performed directly over the nonlinear surrogate using projected gradient descent, and linear and quadratic penalty formulations are empirically compared. The framework is evaluated on two complementary OPF-Gym benchmarks, EcoDispatch and MaxRenewables, which differ in their optimization objectives, network topologies, actuator compositions, and control dimensionalities. Large-scale out-of-sample simulations show substantial reductions in the frequency and severity of constraint violations relative to nominal SAC, Lagrangian-penalty SAC, Recovery RL, and linearized correction baselines; across both benchmarks, the exact-gradient variants substantially reduce physical constraint violations relative to nominal SAC and the linearized correction strategies. In EcoDispatch, these feasibility improvements are additionally achieved while maintaining reward performance close to the underlying SAC controller. The results demonstrate the potential of simulator-driven deployment-time action correction for improving the empirical feasibility of RL-based OPF decisions without retraining the nominal policy or requiring the analytical OPF constraints to be exposed in differentiable form.