Mp17-13 Assessing the Usability of Live Surgical Feedback for Intraoperative Training Improvement

You have accessJournal of UrologyEducation Research I (MP17)1 May 2024MP17-13 ASSESSING THE USABILITY OF LIVE SURGICAL FEEDBACK FOR INTRAOPERATIVE TRAINING IMPROVEMENT J. Everett Knudsen, Runzhuo Ma, Elyssa Y. Wong, Lydia Lin, Eman Dadashian, Luis Medina, Marissa Maas, Randall Lee, Mitchell Goldenberg, Steven Cen, Xiuzhen Huang, and Andrew J. Hung J. Everett KnudsenJ. Everett Knudsen , Runzhuo MaRunzhuo Ma , Elyssa Y. WongElyssa Y. Wong , Lydia LinLydia Lin , Eman DadashianEman Dadashian , Luis MedinaLuis Medina , Marissa MaasMarissa Maas , Randall LeeRandall Lee , Mitchell GoldenbergMitchell Goldenberg , Steven CenSteven Cen , Xiuzhen HuangXiuzhen Huang , and Andrew J. HungAndrew J. Hung View All Author Informationhttps://doi.org/10.1097/01.JU.0001008628.15460.84.13AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Surgical training requires trainees to receive continuous intraoperative feedback from attending surgeons, but it is unknown how feedback quality affects a trainee's ability to effectively change their behavior in the OR. This study aims to quantifiably assess feedback quality and determine its association with intraoperative behavioral outcomes. METHODS: Building on previous work where 3,711 feedback video clips from robotic urological cases were transcribed and categorized, we selected 157 clips where the trainer gave feedback, the trainee changed their behavior, and the trainer appraised the outcome as optimal (approval N=81) or suboptimal (disapproval N=76) (Figure 1A). We then developed a 5-point, 5-question Feedback-Specific System Usability Scale (FS-SUS) from the validated SUS (Figure 1B). Three groups of raters of different surgical experience levels (Figure 1B) were recruited to retrospectively apply FS-SUS to the feedback clips as if they were the trainee receiving intraoperative feedback. FS-SUS scores were calculated by standard methodology and compared between outcomes by hierarchical modeling. RESULTS: Through variance component analysis, we observed a>95% variance contribution in feedback usability scores from individual variation among trainers (Figure 1D). For all raters considered, the median FS-SUS was statistically significantly higher for the optimal outcome (median IQR: 75 60-95) as compared to the suboptimal outcome (70 55-90, p<0.001). A similar pattern holds true when examining raters of different experience levels: expert (70 60-80 vs. 65 55-75, p=0.05), intermediate (70 55-85 vs. 65 50-80, p=0.027), and novice (90 (80-100) vs. 85 (52.5-100), p=0.002) (interaction p=0.39). CONCLUSIONS: As we have shown, there is a high degree of variability in feedback quality observed for all individual trainers. Additionally, higher quality feedback is perceptible by raters of varying experience levels, and higher quality feedback is associated with better trainee behavioral outcomes. This indicates a need to improve the quality of interoperative feedback delivered to surgical trainees at all training levels. Download PPT Source of Funding: None © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e296 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information J. Everett Knudsen More articles by this author Runzhuo Ma More articles by this author Elyssa Y. Wong More articles by this author Lydia Lin More articles by this author Eman Dadashian More articles by this author Luis Medina More articles by this author Marissa Maas More articles by this author Randall Lee More articles by this author Mitchell Goldenberg More articles by this author Steven Cen More articles by this author Xiuzhen Huang More articles by this author Andrew J. Hung More articles by this author Expand All Advertisement PDF downloadLoading ...