Artificial Intelligence Can Accurately Detect Tools Used During Colonoscopy: Another Step Forward Toward Autonomous Report Writing: Presidential Poster Award

Introduction: Artificial intelligence (AI) using convolutional deep learning techniques is showing tremendous potential in enhancing the practice of colonoscopy. Early studies showing that AI can assist colonoscopists with detection and classification of colon polyps are already underway. Many believe that AI will have a large role in “autonomous documentation” that is assist in the writing of the procedural report saving time and effort. We are currently building an AI system for colonoscopy whose goal will be to document a complete endoscopic report based on the Al continually “watching” what the proceduralist is doing. In this study, we designed an AI algorithm that learned different endoscopic tools that are used during colonoscopy and then tested the algorithm's ability to recognize these tools. Methods: From a database of over 180,000 high definition images derived from colonoscopy procedures at our institution, approximately 56,000 were found to have devices depicted. Using these images, a convolutional neural network was trained using a pre-trained model and trained to classify images with tools into groups for snare, forcep, APC catheter, cuff/cap, clips, dilation balloon, loops, injection needle and rothnet. This process is known as “transfer learning.” 80% of the images were separated to train the model and the remaining 20% were used to validate the model's accuracy, sensitivity, and specificity. Results: For identification of the presence of any tool, the algorithm demonstrated an accuracy of 0.97, area under the curve (AUC) of 0.96, sensitivity of 0.95, and specificity of 0.97. For clips: accuracy was 0.98, AUC was 0.98, sensitivity was 0.99, and specificity was 0.96. For cuff/cap: accuracy was 0.99, AUC was 0.99, sensitivity was 0.99, and specificity was 0.98. For APC catheters, accuracy was 0.996, AUC was 0.96, sensitivity was 0.998, and specificity was 0.92. For snares: accuracy was 0.97, AUC was 0.96, sensitivity was 0.99, and specificity was 0.90. For forceps, accuracy was 0.94, AUC was 0.93, sensitivity was 0.96, and specificity was 0.90. The algorithm was capable of processing each frame at around 4 milliseconds per frame. Conclusion: We successfully developed an artificial intelligence algorithm that can accurately predict endoscopic tools being used during colonoscopy using still images. The next step will include validating this algorithm as a video overlay of live colonoscopy procedures to assist in autonomous documentation of the endoscopic report.1075_A Figure 1. Confusion matrix for the detection of tool presence1075_B Figure 2. Area under the curve for the detection of tool presence