Physics-Informed Koopman Neural Operator for Augmented Dynamics Visual Servoing of Multirotors

This paper introduces a Physics-Informed Koopman Neural Operator (PI-KNO) for augmented dynamics visual servoing of multirotors that integrates Koopman operator theory with neural networks. The proposed method establishes a structured learning framework that effectively captures complex system dynamics while embedding physics-based priors. Unlike fully data-driven approaches, PI-KNO improves generalization and minimizes reliance on extensive real-world training data by employing a hybrid loss function that combines physics-informed constraints with real-time observations. The learned model is incorporated into a monotonically weighted nonlinear model predictive control (NMPC) framework, ensuring precise trajectory tracking while adhering to state and input constraints. Experimental results demonstrate that PI-KNO reduces training time by 16.81% and enhances tracking accuracy by 19.56% compared to conventional Data-Driven Koopman Neural Operators (DD-KNO) and Physics-Informed Neural Networks (PINN). Additionally, under 70.83% uncertainty in camera parameters, PI-KNO achieves 10.6% and 22.2% lower tracking errors than PINN and DD-KNO, respectively. These findings underscore the robustness and efficiency of the proposed approach for real-time multirotor visual servoing applications. Note to Practitioners - This work presents an implementation oriented view of a learning based predictive controller for aerial robots designed to operate within the limits of onboard computation and sensing. The proposed PI-KNO approach combines physical modeling and data driven learning to create an accurate and stable dynamic predictor suitable for real time deployment. The architecture is distributed between an NVIDIA Jetson and a Pixhawk. The Jetson runs the vision front end, state estimator, PI-KNO predictor, and NMPC optimizer. The estimator fuses visual and inertial data to provide position, velocity, and attitude states, while the NMPC computes high level motion commands at about 30Hz using the PI-KNO rollout. The Pixhawk executes inner rate loops at 250Hz through the native autopilot for attitude stabilization and motor mixing. Communication between the Jetson and Pixhawk uses standard MAVLink setpoints, requiring no firmware modification. The training workflow begins with a nominal visual servoing multirotor model and limited flight data, from which the operator is trained offline using a physics-informed regularizer. At runtime, the estimator updates the state, PI-KNO predicts short horizon dynamics, the NMPC optimizes control sequences, and the Pixhawk executes the commands. This setup achieves reliable tracking across trajectories, maintains physical consistency under uncertainty, and fits within the computational limits of embedded hardware.