End-to-End Partial Convolutions Neural Networks for Dunhuang Grottoes Wall-Painting Restoration

In this paper, we focus on training a deep neural network to in-paint and restore the historical painting of Dunhuang Grottoes. Dunhuang Grottoes is more than 1000 years old and the wall-painting on the grottoes has suffered from various deterioration. The ground truth does not exist either. Furthermore, learning the style of the artists is not straight forward because the wall-paintings are created by thousands of artists over more 400-500 years. As the very first attempt to solve this problem, we propose an end-to-end image restoration model for Dunhuang wall-painting. The end-to-end image restoration model employ U-net with partially convoluational layers to construct, which is capable in restoring non-rigid deteriorated content given a loss content mask and a wall-painting image. To learn the various artists style from real data, the training set and validation set are collected by using a zooming-in-like and random cropping approach on the digital RGB images photographed on the healthy Grotto-painting. We also synthesize the deteriorated paintings from real data. To ensure the synthetic content in the masked region is consistent to the ground truths in term of texture, colors, artistic style and free of unnecessary noises, the loss function is in a hybrid form that comprises transition variation loss, content loss and style loss. Our contributions are of three folds: 1) proposed using partial convolutional U-net in restoring wall-paintings; 2) the method is tested in restoring highly non-rigid and irregular deteriorated regions; 3) two types of masks are designed for simulating deteriorations and experimental results are satisfactory.