Key result
Multimodal transformer architecture outperforms state-of-the-art models with ~90% accuracy for real-time surgical gesture prediction.
Why the study?
Real-time recognition and prediction of surgical activities are fundamental to advancing safety and autonomy in robot-assisted surgery.
A novel multimodal transformer architecture effectively fuses kinematic and video features to achieve state-of-the-art accuracy and real-time performance in surgical gesture prediction.
May support real-time surgical AI tools; leaves open clinical validation and outcome impact.
Real-time recognition and prediction of surgical activities are fundamental to advancing safety and autonomy in robot-assisted surgery. This paper presents a multimodal transformer architecture for real-time recognition and prediction of surgical gestures and trajectories based on short segments of kinematic and video data. We conduct an ablation study to evaluate the impact of fusing different input modalities and their representations on gesture recognition and prediction performance. We perform an end-to-end assessment of the proposed architecture using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset. Our model outperforms the state-of-the-art (SOTA) with 89.5% accuracy for gesture prediction through effective fusion of kinematic features with spatial and contextual video features. It achieves the real-time performance of 1.1-1.3ms for processing a 1-second input window by relying on a computationally efficient model.
No takes yet. Share an insight, caveat, or question.
Weerasinghe et al. (2024) studied Robot-assisted surgery. Multimodal transformer architecture vs. State-of-the-art (SOTA) models was evaluated on Gesture prediction accuracy. A multimodal transformer architecture outperformed state-of-the-art models with 89.5% accuracy for surgical gesture prediction while achieving real-time processing speeds of 1.1-1.3ms.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: