Synthesizing Interacting Model‐Based Optimal Control and Model‐Free Learning Approaches for Nonlinear Systems

ABSTRACT In this paper, we consider the optimal control problem for an unknown continuous‐time nonlinear system, and present a framework that integrates model‐based and model‐free methods to solve it. Each approach offers distinct advantages: model‐based techniques provide offline synthesis and data efficiency, while model‐free procedures excel at accuracy and online learning capabilities. To harness the strengths of both approaches, we propose a control design that starts with a controller derived from an available linear model, and is subsequently augmented with a learned (model‐free) controller component. Specifically, we derive an optimality‐based condition that determines the switching instant from the purely model‐based controller to the composite controller, which includes both model‐based and model‐free components. Importantly, we dispense with the assumption of a known degree of misalignment between the model and the system; the switching condition accounts for real‐time model mismatch, ensuring that a more inaccurate model results in a faster switching time. Finally, to obtain the model‐free component in the composite controller, we employ an off‐policy reinforcement learning (RL) algorithm that uses trajectory data to learn the optimal control augmentation. Simulations demonstrate the efficacy of the proposed control framework.