Memory Uncertainty Learning for Real-World Single Image Deraining

Single image deraining has witnessed dramatic improvements by training deep neural networks on large-scale synthetic data. However, due to the discrepancy between authentic and synthetic rain images, it is challenging to directly extend existing methods to real-world scenes. To address this issue, we propose a memory-uncertainty guided semi-supervised method to learn rain properties simultaneously from synthetic and real data. The key aspect is developing a stochastic memory network that is equipped with memory modules to record prototypical rain patterns. The memory modules are updated in a self-supervised way, allowing the network to comprehensively capture rainy styles without the need for clean labels. The memory items are read stochastically according to their similarities with rain representations, leading to diverse predictions and efficient uncertainty estimation. Furthermore, we present an uncertainty-aware self-training mechanism to transfer knowledge from supervised deraining to unsupervised cases. An additional target network is adopted to produce pseudo-labels for unlabeled data, of which the incorrect ones are rectified by uncertainty estimates. Finally, we construct a new large-scale image deraining dataset of 10.2 k real rain images, significantly improving the diversity of real rain scenes. Experiments show that our method achieves more appealing results for real-world rain removal than recent state-of-the-art methods.