Pd27-04 Human-AI Collaboration for Unsupervised Categorization of Live Surgical Feedback

You have accessJournal of UrologySurgical Technology & Simulation: Artificial Intelligence II (PD27)1 May 2024PD27-04 HUMAN-AI COLLABORATION FOR UNSUPERVISED CATEGORIZATION OF LIVE SURGICAL FEEDBACK Rafal Kocielnik, Cherine H. Yang, Runzhuo Ma, Elyssa Y. Wong, Timothy N. Chu, Dani Kiyasseh, Lydia Lin, Eman Dadashian, Jiayun Wang, Xiuzhen Huang, Animashree Anandkumar, and Andrew J. Hung Rafal KocielnikRafal Kocielnik , Cherine H. YangCherine H. Yang , Runzhuo MaRunzhuo Ma , Elyssa Y. WongElyssa Y. Wong , Timothy N. ChuTimothy N. Chu , Dani KiyassehDani Kiyasseh , Lydia LinLydia Lin , Eman DadashianEman Dadashian , Jiayun WangJiayun Wang , Xiuzhen HuangXiuzhen Huang , Animashree AnandkumarAnimashree Anandkumar , and Andrew J. HungAndrew J. Hung View All Author Informationhttps://doi.org/10.1097/01.JU.0001008580.58088.27.04AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Verbal feedback in the operating room is vital for safe surgeries and training. Analyzing the characteristics of this feedback is key to enhancing surgical training. We use unsupervised machine learning (Topic Modeling) with human input to automatically categorize feedback components from surgical transcripts. METHODS: We use a dataset of 3912 instances of surgical feedback delivered over the course of 31 surgical cases covering 6 types of procedures in the context of robot-assisted surgery (RAS). Feedback was defined as any dialogue intended to modify trainee thinking or behavior provided by a trainer. Feedback quotations were transcribed. We applied an automated unsupervised topic modeling technique called BERTopic, which extracts a representation of text (embedding) using a pre-trained large language model–BERT. This representation captures the semantic "meaning" of the text and groups feedback instances with similar meaning into the same Topic. The initial number of Topics was selected using the coherence score metric and further refined based on the clinicians' guidance (Figure 1A). Discovered Topics were evaluated by two clinicians (Figure 1B) in terms of "clinical clarity" defined as "meaningfulness for clinical practice" following two rounds of evaluation and refinement. Clinicians' agreement was measured with intra-class correlation (ICC). RESULTS: Initial automated topic modeling resulted in the discovery of 28 topics. These automatically extracted Topics (Figure 1B) received a human-derived average clinical clarity score of 3.14 (SD=1.36). After clinicians' input, the topics were consolidated into 20 which scored significantly higher on clinical clarity 3.98, (SD=0.92, p<0.05). Clinical clarity scores exhibited high agreement among raters (ICC=0.78, p<0.01). The top-scoring Topics related to aspects of high clinical relevance such as "Handling and Positioning of (tissue)" (Topic 17, Figure 1C), "(Tissue) Layer Depth Assessment and Correction" (Topic 19), and "Fat Dissection and Exposure" (Topic 18). CONCLUSIONS: We demonstrated the utility of using topic modeling and human input to iteratively categorize clinically relevant surgical feedback delivered in RAS. These findings support the development of human-AI collaboration to ensure the clinical relevance of AI-driven efforts in surgical training. Download PPT Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e551 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Rafal Kocielnik More articles by this author Cherine H. Yang More articles by this author Runzhuo Ma More articles by this author Elyssa Y. Wong More articles by this author Timothy N. Chu More articles by this author Dani Kiyasseh More articles by this author Lydia Lin More articles by this author Eman Dadashian More articles by this author Jiayun Wang More articles by this author Xiuzhen Huang More articles by this author Animashree Anandkumar More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...