Combining Pruning Strategies to Generate Efficient Inpainting Networks

Image Inpainting methods based on Neural Networks have shown an impressive improvement in the past few years, achieving ever more convincing results. However, these improvements have come at the cost of larger and more computationally intensive models, which demand powerful hardware for their execution. In this paper we introduce a high-quality and lightweight image inpainting architecture, based on LaMa’s frequency reconstruction network, using Unconstrained Blueprint Separable Convolutions, in combination with Relevant Channel Pruning and our technique for Residual Layer Pruning. By using Unconstrained Blueprint Separable Convolution, we manage to efficiently reduce the number of model weights by decorrelating the learned knowledge. We also observe that shallower residual layers within the network’s bottleneck are the main contribution to the quality of the final output of the network. Furthermore, we observe that weights with a larger L1 norm in the decoder represent the reconstruction of lower-frequency details in the image, such as color and coarse structures, while smaller L1 norm weights are responsible for high-frequency details in the image, such as edges and finer textures. By pruning these components in the network and performing fine-tuning on the remaining weights, the performance of the network can be improved while reducing the required resources. Our method can be applied to encoder-decoder architectures, and manages to reduce the model complexity while maintaining a competitive result compared to other models in the literature. The proposed architecture reduces by 72.89% the number of model parameters, and 65.80% in model FLOPs compared to LaMa, and compares favorably to state-of-the-art models when measuring FID and LPIPS of models of similar number of parameters and complexity.