Reinforcement Active Modeling for Flexible Needle Shape Prediction in Multilayer Tissues

The complex interactions between flexible needles and tissues present significant challenges in predicting the needle shape during the puncture procedure. In particular, the accurate prediction of flexible needle shape during insertion into complex multilayer tissues, especially when measurement feedback involves non-Gaussian noise, remains an open problem. In this article, we develop a novel reinforcement learning-based active modeling scheme to predict the deflection of the robotic flexible needle. First, the active modeling scheme is constructed by deriving an extended Kalman filter under the maximum correntropy criterion to enhance insensitivity to non-Gaussian noise. Subsequently, based on this scheme, the reinforcement active modeling (RAM) framework is built by incorporating reinforcement learning to compensate for the modeling residuals. Specifically, the theoretical convergence of the proposed scheme is proved by using the Banach fixed-point theorem, thereby ensuring the reliability of needle shape prediction. Finally, a series of comparative experiments is carried out on a self-built robotic flexible needle. The experimental results demonstrate the superior performance of the proposed deflection predictor. Under non-Gaussian noise conditions, the proposed RAM scheme achieves a generalization prediction error reduction of 46.4% in RMSE and over 76.1% in Var during insertion into unknown multilayer tissue.