Segmentation of Image Observations for Articulated Endoscope Control

Abstract We present a method to reduce the sim-to-real gap for down-stream reinforcement-learning approaches that are trained in simulation. Our pipeline converts endoscopic videos into simplified segmentation masks without any model retraining. Using the zero-shot capability of the Segment Anything Model (SAM), a lightweight HSV filter and SORT tracking, our method produces consistent semantic masks for a pickand- place task with an articulated endoscope. We quantitatively show the importance of each of these components. Unlike typical sim-to-real approaches that render synthetic data more realistic, we make real footage "look synthetic". Experiments on a physical setup demonstrate promising segmentation masks even under partial occlusion, motion blur and lighting variations.