Artificial intelligence-supported segmentation of cardiac anatomy in open-heart surgery videos

Abstract Accurate identification of anatomical structures is essential for safe congenital cardiac surgery and for the development of assistive robotic systems. However, scalable annotation of open-heart surgical video remains a major challenge due to dynamic tissue motion, occlusion, and anatomical variability. The aim of this study was to develop and evaluate a human-in-the-loop segmentation and tracking pipeline for congenital open-heart surgery videos. A dataset of 72 annotated video clips comprising 27,461 frames was created from routine recordings of congenital cardiac surgery. Independent tracking evaluation was performed on 6 video clips (2,400 frames) from three surgical cases that were not used for training or validation. A hybrid framework combining fine-tuned Segment Anything Model 2 (SAM 2) for propagation-based tracking and YOLOv11 for detection-based monitoring was implemented and evaluated. Fine-tuning improved temporal tracking performance compared with the pretrained model, increasing the mean IoU from 82.1% to 92.2% and the proportion of frames with IoU ≥ 90% from 48.2% to 72.3% on the independent evaluation dataset. A graphical user interface enabled efficient dataset creation, reducing annotation time by approximately 40-fold compared with manual tracing. This study demonstrates that domain-adapted foundation-model tracking can provide robust anatomical tracking in a dynamic open surgical environment and offers a scalable framework for future assistive and robotic cardiac applications.