Diminished reality using semantic segmentation and generative adversarial network for landscape assessment: evaluation of image inpainting according to colour vision

Abstract The objective of this research is to develop a method to detect and virtually remove representations of existing buildings from a video stream in real-time for the purpose of visualizing a future scenario without these buildings. This is done by using semantic segmentation, which eliminates the need to create three-dimensional models of the buildings and the surrounding scenery, and a generative adversarial network (GAN), a deep learning method for generating images. Real-time communication between devices enables users to utilize only portable devices equipped with a camera to visualize the future landscape onsite. As verification of the proposed method’s usefulness, we evaluated the complementation accuracy of the GAN and real-time performance of the entire method. The results indicated that the process is completed accurately when the area to be complemented is less than 15% of the view and that the process runs at 5.71 fps. The proposed method enables users to understand intuitively the future landscape and contributes to reducing the time and cost for building consensus.