Deep-learning-based endoscopic single-shot fringe projection profilometry

SignificanceConventional fringe projection profilometry (FPP) requires multiple image acquisitions and therefore long acquisition times that make it slow for high-speed dynamic measurements. We propose and demonstrate a deep-learning-based single-shot FPP system utilizing a single endoscope for surgical guidance.AimWe aim to achieve real-time depth map generation of target tissues with high accuracy for robotic surgical guidance.ApproachWe proposed an endoscopic single-shot FPP system based on a deep learning network to generate real-time accurate tissue depth maps for surgical guidance. The system utilizes a dual-channel endoscope, where one channel projects fringe patterns from a projector and the other channel collects images using a camera. In addition, we developed a data synthesis method to generate a large number of diverse training datasets. The network consists of MaskNet, which segments the tissue from the background, and DepthNet, which predicts the depth map of the image. The results from both networks are combined to generate the final depth map.ResultsWe tested our algorithm using fringe patterns with different frequencies and found that the optimal frequency for single-shot FPP in our setup is 20 Hz. The algorithm has been tested on both synthetic and experimental data, achieving a maximum depth prediction error of ∼2 mm and a processing time of about 12.75 ms per frame.ConclusionA deep-learning-based single-shot FPP endoscopic system was shown to be highly effective in real-time depth map generation with millimeter-scale error. Implementing such a system has the potential to improve the reliability of image-guided robotic surgery.