OS-NeRF: Generalizable Novel View Synthesis for Occluded Open-Surgical Scenes

The process of video recording open surgical procedures is a critical aspect of medical practice, as it enables physicians to retrospectively analyze and assess the surgery. The synthesis of a novel view of surgical procedures from recorded footage is a vital task, facilitating the review of these procedures from multiple angles. Although employing neural radiance fields (NeRF) may seem a straightforward approach, the application of standard NeRF models to medical environments often results in suboptimal rendering quality. This issue predominantly arises due to the significant occlusions present in open surgical videos and the constraints imposed by the limited number of cameras available, further exacerbated by the unique environmental restrictions of the operating room. Through the adoption of a learning framework for novel view rendering, derived from both natural and open surgical scenes using a generalizable NeRF representation, our methodology offers the capacity to render new perspectives from a constrained set of source images. Furthermore, we implement an automatic occlusion mask estimation to exclude obscured pixels during the training phase. The efficacy of our proposed approach has been validated through comparative analyses with other robust training configurations, utilizing real-world video data recorded at Keio University School of Medicine. Additionally, our findings demonstrate the practical effectiveness of the proposed occlusion mask.