Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

Due to the limitations of satellite sensors, we can only obtain MS images and PAN images separately. The focus of our attention is to utilize the pan-sharpening method to generate the high-resolution multispectral (HRMS) images. In this paper, we proposed the dual-branch multi-scale fusion network (DMFN), which based on the Spatial-Spectral Transformer to comprehensively capture the information contained in MS images and PAN images at different scales. The architecture of our network consists of three parts: during the feature extraction and image fusion stage, we first independently apply upscaling and downscaling operations to the MS and PAN images. Subsequently, we concatenate the images from the two distinct branches and input them into the shallow feature extraction module individually. And then we input them into our Adaptive Feature Extraction Block (AFEB) to further extract the crucial details of the images using the attention mechanism. The images at various scales in different branches are then passed through three (Spectral Transformer) SPET and three (Spatial Transformer) SPAT modules to perform a comprehensive extraction of both spatial and spectral characteristics. Finally, the Residual Local Feature Module (RLFM) is utilized during the image reconstruction part to deeply extract intricate information from the images and obtain the final HRMS fused image. We have conducted both simulated and real experiments on the benchmark datasets QB and WV2. The final qualitative and quantitative comparative results demonstrate that our innovative method outperforms the current SOTA methods.