Near-Field Neural Rendering Guided by Single-Shot Photometric Stereo

We present a novel near-field neural rendering approach that combines single-shot RGB photometric stereo and SDF-based Neural Rendering. Photometric stereo-based cues guide the neural rendering of 3D meshes. Recent studies have shown that SDF-based neural implicit surface reconstruction methods can produce smoother, more comprehensive 3D reconstructions. However, their performance tends to decline when capturing fine details in complex near-field images due to the ambiguity in RGB reconstruction loss. For instance, 3D endoscopy, characterized by near-field sparse views and complex surfaces, poses considerable challenges for existing volumetric SDF methods. Motivated by advancements in near-field photometric stereo, our work explores the utilization of these cues to enhance neural implicit surface reconstruction from diverse perspectives. To simplify the acquisition of photometric stereo images in endoscopy setups, we employ a strategy that involves single-shot RGB photometric stereo capture for each view. Utilizing a learning-based near-field photometric stereo network, we extract depth and normals for each view. These cues contribute to improved performance when dealing with very near-field natural objects as well as endoscopic scenes. We have extensively tested this approach on various rendered and real near-field datasets, and it consistently outperforms existing photometric NeRF techniques, especially when dealing with near-field multi-view visuals.