PSNet: Towards Efficient Image Restoration With Self-Attention

Image restoration aims to recover a clean image from various degradations, e.g., haze, snow, and blur, playing an important role in robot vision, autonomous vehicles, and medical imaging. Recently, the use of Transformer has witnessed a significant improvement in multifarious image restoration tasks. However, despite a few remedies to reduce the quadratic complexity of self-attention, these approaches are still impractical for real-world applications, which need high efficiency and speed. To ameliorate this issue, we propose an efficient framework for image restoration based on self-attention. To this end, we combine the strengths of patch-based and strip-based self-attention units to improve efficiency. More specifically, we apply self-attention of different operation scales to features of different resolutions, i.e., we adopt a relatively smaller region for self-attention on high-resolution features while a larger region for low-restoration features. In addition, instead of using global self-attention in each partitioned region, we leverage a strip-based version for low complexity. To further improve efficiency, we insert our design into a U-shaped CNN network to establish our framework, dubbed PSNet. Extensive experiments demonstrate that our network receives state-of-the-art performance on five representative image restoration tasks with low computational complexity and high speed, including single-image defocus deblurring, image dehazing, image motion deblurring, image desnowing, and image denoising.