Real-Time Inpainting for Stereoscopic Diminished Reality

Inpainting is the process of reconstructing or replacing regions of an image by utilizing contextual information. Algorithmic approaches for inpaiting solve an optimization problem by minimizing consistency error along multiple samples of an image and the gaps to inpaint. Another method is the usage of ML, to learn image features, and extrapolate it over masked areas of the image. Recent research shows that appropriately designed ML-Models achieve high quality inpainting results for even high-resolution images while avoiding the high memory demands typically implied by them. In this paper, the method of ML-based inpainting is extended to stereoscopic real-time video feeds. It presents a modified model for contextual residual aggregation, and a framework for utilizing it for the purpose of inpainting stereoscopic real-time video feeds. Aspects of the real-time rendered scene are then masked by chroma-keying to inpaint a variety of areas. The results are then tested for their structural integrity by calculating multiple similarity scores, comparing the result with the ground truth to give a final estimate for the overall inpainting quality.