Mixing Self-Attention and Convolution: A Unified Framework for Multisource Remote Sensing Data Classification

Convolution and self-attention are two powerful techniques for multi-source remote sensing (RS) data fusion that have been widely adopted in Earth observation tasks. However, Convolutional Neural Networks (CNNs) are inadequate for fully mining contextual information and representing the sequence attributes of spectral signatures. Additionally, the specific self-attention mechanism often comes with high computation costs, which hinders its application in the field of RS. To overcome the above limitations, this paper proposes a unified framework called “Mixing Self-Attention and Convolution Network" for comprehensive feature extraction and efficient feature fusion. First, the proposed MACN utilizes two adaptive CNN encoders (ACEs) to extract shallow convolutional features from multi-source RS data. Secondly, taking the complexity and varying scales of RS data into account, the proposed mixing self-attention and convolution transformer (MACT) layer achieves local and global multiscale perception through an elegant integration of self-attention and convolution. MACT can extract abundant spatial and high-dimensional information (e.g., spectral and elevation information) while maintaining minimal computational overhead compared to pure convolution or self-attention counterparts. Finally, a multi-source cross-guided fusion (MCGF) module is designed to achieve deep fusion of multi-source RS data features. MCGF utilizes a carefully designed cross-modal attention mechanism to capture the interaction between multi-source data and aggregate contextual information. Extensive tests on six public RS datasets have shown that our method outperforms other multi-source fusion models, delivering state-of-the-art results on multiple RS data fusion tasks without specific tuning. The source code of the proposed method will be available publicly at https://github.com/like413/MACN.