Mp07-20 the Role of Unsupervised Machine Learning in Robotic Surgery Skill Assessment

You have accessJournal of UrologySurgical Technology & Simulation: Artificial Intelligence I (MP07)1 May 2024MP07-20 THE ROLE OF UNSUPERVISED MACHINE LEARNING IN ROBOTIC SURGERY SKILL ASSESSMENT Katherina Y. Chen, Kay Hutchinson, Homa Alemzadeh, and Noah S. Schenkman Katherina Y. ChenKatherina Y. Chen , Kay HutchinsonKay Hutchinson , Homa AlemzadehHoma Alemzadeh , and Noah S. SchenkmanNoah S. Schenkman View All Author Informationhttps://doi.org/10.1097/01.JU.0001008728.41882.d7.20AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Assessment of a surgical trainee's skills has traditionally been subjective evaluation, which can be variable and have poor inter-rater reliability. However, the growing number of robotic-assisted surgeries has created a wealth of data and an opportunity for computer-driven analysis of surgical skills, which can provide an objective and quantifiable evaluation. However, supervised machine learning requires a labelled dataset, which can be expensive and time-intensive to produce. The objective of this study was to evaluate the use of unsupervised machine learning using clustering algorithms to distinguish between surgeon skill levels. METHODS: This study utilized the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), which includes video data from eight surgeons of varying skill levels who performed suturing and needle passing in a dry lab as well as gesture labels. K-modes and k-means clustering algorithms and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) were run for each gesture of each surgical task. The Silhouette Score (SS) was used to quantify the performance of the clustering algorithms and the Normalized Mutual Information (NMI) score was used to evaluate the cluster assignments compared to ground truth experience level labels. RESULTS: DBSCAN produced the highest SS across all the gestures of each task, ranging from 0.50 to 0.78 for suturing and 0.56 to 0.86 for needle passing. However, DBSCAN also produced the lowest NMI scores in relation to the surgeons' self-reported skill level, ranging from 0.01 to 0.07 for suturing and 0.01 to 0.10 for needle passing. K-means clustering with a naïve selection of 3 for k produced the highest NMI scores, ranging from 0.18 to 0.84 for suturing and 0.22 to 0.48 for needle passing. Comprehensive results are shown in Table 1. CONCLUSIONS: Unsupervised machine learning, and in particular k-means clustering, has potential in the objective assessment of robotic surgical skills. The high SS of DBSCAN along with its low NMI scores, point to the need for further refinement of these algorithms to enhance their evaluative precision. As the availability of labeled video data for supervised learning remains scarce, unsupervised learning stands out as a viable alternative, meriting further exploration and development in future urologic research. Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e113 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Katherina Y. Chen More articles by this author Kay Hutchinson More articles by this author Homa Alemzadeh More articles by this author Noah S. Schenkman More articles by this author Expand All Advertisement PDF downloadLoading ...