MedSAM-Flow: A Deep Fusion Methodology for Robust Realtime Tracking of Objects in Laparoscopic Videos

Artificial intelligence (AI) has revolutionized various aspects of medicine, particularly in surgical procedures, by offering advanced tools for analysis and decision-making 1- 3. Real-time video-based surgical instrument segmentation holds paramount importance in operating rooms (ORs), reshaping surgical practices and patient care standards. The ability to accurately segment surgical instruments in real-time enables surgeons to receive instantaneous visual feedback, facilitating precise adjustments during procedures. Such immediate feedback not only enhances surgical technique refinements but also allows for on-the-spot skill assessment, empowering surgeons to continually improve their performance. Moreover, real-time segmentation facilitates comprehensive performance evaluation in gastrointestinal surgery, contributing to better patient outcomes through optimized surgical processes and reduced risks 4,5. This paper introduces a novel approach to address the critical need for real-time surgical instrument segmentation in ORs using intraoperative videos. The proposed method builds upon MedSAM model 1, fine-tuned specifically for medical image segmentation, to achieve accurate real-time segmentation in surgical videos. By leveraging Optical Flow 6 and Bezier 7 methods, this approach overcomes the limitation of MedSAM's slow execution time, ensuring fast and efficient segmentation in video streams. Furthermore, the methodology presented in this paper not only enables real- time segmentation but also offers smooth instrument tracking across consecutive frames, enhancing the overall efficiency of surgical procedures. With its potential to significantly impact surgical practices, this paper contributes to the advancement of real-time video-based surgical instrument segmentation and its applications in gastrointestinal surgery. In addition to its immediate benefits for surgical procedures, the proposed method sets the stage for future developments in AI-driven surgical interventions. By addressing the need for real-time segmentation in ORs and providing a framework for further refinement and customization, this paper paves the way for the development of tailored AI solutions that cater to specific surgical needs. Through ongoing collaboration with expert surgeons and the continued exploration of advanced AI techniques, the potential for improving patient care standards and advancing surgical practices remains promising. The subsequent sections of this paper will delve into the materials and methods employed, followed by a presentation of the results obtained from the proposed approach. The discussion section will analyze the implications of these findings, while also exploring future directions for research and application in the field of real-time surgical instrument segmentation.