Key result
A multi-modal deep learning model achieved performance comparable to expert human raters for assessing surgical knot-tying skills, demonstrating the best overall performance among the tested models.
Why the study?
Objective assessment of technical skill is a growing but resource-heavy element of surgical education, motivating the development of automated surgical skill assessment.
Does a multi-modal deep learning model achieve comparable performance to expert human raters in assessing surgical knot-tying skills in surgical trainees and faculty?
Does a multi-modal deep learning model achieve comparable performance to expert human raters in assessing surgical knot-tying skills in surgical trainees and faculty?
Multi-modal deep learning models can achieve state-of-the-art performance comparable to expert human raters in objective surgical skill assessment, potentially reducing the burden on training faculty.
Answers are grounded in this paper and cite their sources.
May reduce faculty burden in surgical training; leaves open prospective validation before adoption.
This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.
No takes yet. Share an insight, caveat, or question.
Kasa et al. (2022) studied Surgical skill assessment (n=72). Multi-modal deep learning model vs. Expert human raters and unimodal deep learning models was evaluated on Skill assessment performance measured using mean squared error (MSE) and intraclass correlation coefficient (ICC) on the OSATS GRS. A multi-modal deep learning model achieved performance comparable to expert human raters for assessing surgical knot-tying skills, demonstrating the best overall performance among the tested models.
Synapse has enriched 10 related papers. Consider them for comparative context: