AI-assisted 3D reconstruction from routine gastroscopy: a technical feasibility study

Due to the limited field of view of endoscopic cameras and the lack of depth perception, it is challenging for diagnosticians to accurately perceive the location and three-dimensional structure of lesions within the entire stomach. For conditions such as early-stage gastric tumors or endophytic submucosal lesions, which cannot be directly observed during laparoscopy, three-dimensional localization of the lesion becomes particularly critical for surgical planning. This retrospective, offline reconstruction study presents a Dense-UNet-based approach for three-dimensional reconstruction of gastroscopy images. A densely connected convolutional neural network (Dense-UNet)-based matching model for gastroscopic images was built and trained. Gastroscopic examination videos from eight patients diagnosed with superficial gastritis were exported, and key frames were extracted to create a gastroscopic image dataset. The gastroscopic images were fed into the model to obtain matching results, which were subsequently applied to the three-dimensional reconstruction of the gastroscopic images. The 3D point cloud generated by the experiment contains the position and color information of all three-dimensional points, as well as the camera pose information of all successfully reconstructed gastroscopy images. Finally, a three-dimensional model of the stomach is generated. In the generated three-dimensional model, all reconstructed gastroscopy images can be directly switched to the corresponding scene from the same perspective and viewed freely. This method improves the number of three-dimensional points, which is more than 2.93 times higher than traditional methods with the number of reconstructed images increasing by 46.7%. This method can effectively increase the number of reconstructed images and 3D points in 3D point clouds, improving the stability and integrity of the reconstruction results. This method successfully reconstructed gastroscopy images in 3D, making it easier for diagnostic physicians to perceive the three-dimensional structure of real gastroscopy image scenes, and providing reference for clinical decision-making.