Shared Spatiotemporal Multitask Learning for Simultaneous Surgical Gesture Recognition and Skill Assessment from Minimally Invasive Surgery Videos

The automatic recognition of surgical gestures and the assessment of surgical skills are essential for improving patient safety and optimizing surgical training processes. In this paper, a shared spatiotemporal multitask learning framework for joint surgical gesture recognition and skill assessment from minimally invasive surgery videos is presented. The proposed architecture simultaneously recognizes surgical gestures and estimates the expertise level of the surgeon from surgical video sequences. It is based on a 3D CNN with two parallel branches based on the ResNet18 backbone and incorporates LSTM module to model spatial temporal features from surgical video sequences. In addition, fully connected layers are used to perform surgical gesture classification in one branch and surgical skill level estimation. The experimental evaluation includes surgical gesture recognition and skill assessment using both segmented video clips and full-length surgical videos. The performance of the method is evaluated with the accuracy, precision, recall, and F1-score, for both tasks. These findings suggest that shared spatiotemporal feature learning provides an effective approach for jointly analyzing surgical gestures and skill levels from video.