ACUM And ATUM: Adaptive Multi-pixel Aggregation Upsample Modules for Lightweight Arbitrary Scale Super-Resolution

Abstract While image super-resolution (SR) has made significant breakthroughs with the application of vision transformer (ViT) framework, researchers have primarily focused on enhancing model backbones, ignoring the influence of upsample modules. Mainstream upsample modules predominantly relying on pixshuffle method which lacks spatial and channel interactions among multiple pixels. Moreover, they are limited to specific scale factors (e.g., ×2/×3/×4) rather than accommodating non-integer or asymmetric SR, constraining their reconstructive capabilities. To address these limitations, this paper proposes an innovative adaptive upsample strategy. This strategy aims to effectively aggregate the extracted features and reconstruct high-resolution (HR) image by learning the location and content correlations among feature map pixels. Based on this strategy, we develop two plug-in modules called Adaptive Convolution Upsample Module (ACUM) and Adaptive Transformer Upsample Module (ATUM) for existing lightweight SR networks to perform scale-arbitrary SR. For any low-resolution (LR) images, the two modules can learn to establish the dependencies between each HR image pixel and multiple LR feature pixels to zoom in it with arbitrary scale factors within a single model. Coupling with different kinds of SR backbones, including CNN-based and Vit-based networks, extensive experiments demonstrate that ACUM and ATUM achieves better performance compared to traditional method at specific scale and more efficient at arbitrary scale.