EasyVis2: a real-time multi-view 3D visualization system for laparoscopic surgery training enhanced by a deep neural network YOLOv8-pose

Abstract Minimally invasive laparoscopic surgery often suffers from limited depth perception and constrained visual fields. To address these limitations, we introduce EasyVis2, an enhanced hands-free, real-time 3D visualization system based on the previous EasyVis1 platform. It utilizes a trocar equipped with an array of micro-cameras to provide an expanded field of view and improved 3D perception. This study aims to adapt deep learning-based multi-view pose estimation to enhance instrument tracking and visualization quality while improving computational efficiency. YOLOv8-Pose, a state-of-the-art deep neural network, was integrated into EasyVis2 for 2D pose estimation across multiple views. A customized training dataset was developed to tailor the model to the surgical domain. Multi-view 2D poses were fused to compute 3D poses, enabling real-time surface rendering of instruments. The algorithm is optimized so that real-time performance is achieved using a desktop computer equipped with a GPU. Evaluation was conducted on separate testing sets with ground truth annotations, and results were reported as the mean over testing sets. The proposed system achieved higher 3D reconstruction accuracy and faster processing speed compared to the previous version using the same number of cameras. The retrained adapted YOLOv8-Pose model achieves a 2D pose estimation precision of 96.6% and sensitivity of 95.9%. The system achieved a back-projection error of 3.809 pixels at a processing speed of 12.6 ms per frame. EasyVis2 improves 3D visualization and tracking, validating its potential for intra-operative guidance, surgical training, and future computer-assisted interventions.