Abstract Purpose: Excessive retraction force during a minimally invasive surgery could cause tissue damage, affecting the surgical outcome. The integration of surgical force feedback relies on incorporating force sensors into the instruments, predominantly for robot-assisted surgery. However, such hardware integration is costly and challenging, and such instruments for conventional laparoscopic surgery are not yet widely adopted. We propose to use video as the primary source to objectively estimate tissue retraction force, which presents a promising approach to assess the skill of surgeons while retracting bowel tissue and addressing the current haptic deficiency without the need for special instruments. Methods: We first introduce an experimental setup for the acquisition of a force-vision dataset with conventional laparoscopic instruments retracting a silicone small bowel phantom as an example. A novel data collection procedure is described along with the corresponding force-signal preprocessing methods. Then, we explore the performance of state-of-the-art ResNet and transformer-based computer vision models for force estimation. Two experiments are undertaken to assess feasibility of vision-based force estimation models. Results: Results show that the models generalize across the collected dataset with varying phantom geometries and camera angles. We find ResNet-based models outperforming transformers, with all results matching or surpassing previous works. We further show that all models could perform real-time force estimation with respect to our data collection rate, with 3D ResNet being the fastest. Conclusion: Vision-based force estimation based on the laparoscopic video feed is a promising way toward force estimation of tissue retraction. A novel use of computer vision models has been proposed with a low-cost data collection pipeline. Since the proposed clinical usage does not require customized equipment, seamless integration to the existing surgical framework is possible. The experimental results additionally demonstrate potentials in the approach to pave the way for real-time haptic feedback and quantitative skill assessment.
Wang et al. (Tue,) studied this question.