Optimal adaptive control of methane production in anaerobic digestion via reinforcement learning

This paper investigates the optimal regulation of methane production in a two-step anaerobic digestion (AD) process described by the AM2 model, incorporating Monod kinetics for acidogenesis and Haldane kinetics for methanogenesis. Following steady-state analysis, the nonlinear model is linearized around multiple equilibrium points, including both stable and unstable operating conditions, to obtain local state-space representations for control design. An infinite-horizon linear quadratic regulator (LQR) problem is formulated and solved online using a reinforcement learning approach based on policy iteration. The proposed method estimates the solution of the Riccati equation directly from measured state trajectories, without explicitly using the state matrix during the policy-evaluation step. The approach is applied using two manipulated inputs, namely the dilution rate and the inlet concentration of the second substrate. Numerical simulations evaluate the controller under multiple scenarios, including regulation around different equilibria, comparison with PI and predictive control strategies, and robustness to disturbances. The results indicate that the proposed method achieves local stabilization and improved regulation performance in simulation while reducing explicit reliance on the state matrix during policy evaluation.