MVSTF-Net: multimodal remote sensing image classification based on variable scale convolution and transformer cross fusion strategies

This is the authors' abstract. We don't add key points for this paper.

The integration of hyperspectral image (HSI) and light detection and ranging (LiDAR) has attracted considerable interest in land cover classification. At present, multi-modal remote sensing (RS) image classification methods are difficult to obtain high-quality features from single-scale feature extraction and convolutional neural networks (CNNs) cannot process a large amount of spectral sequence information in HSI, which affects the classification accuracy. In this study, we propose a multimodal RS image classification method based on variable scale and transformer fusion, i.e. MVSTF-Net classification network. During the feature extraction process, the approach is designed to accomplish the integration-based categorization of HSI and LiDAR data. The determination of distinct scale information is facilitated through the utilization of similarity calculation during feature extraction, followed by the selection of appropriate scale features. The problem of low feature quality and high feature dimension in single scale feature extraction is solved by reducing feature dimension and computation. In the process of feature fusion, the two modes are tokenized, and a transformer-based Cross Self-Attention fusion module (CAF) is designed to realize the label feature fusion of the two modes, and effectively fuse the feature information of HSI and LiDAR data. The main contribution of this paper is to propose a new CAF fusion strategy. By cross-fusion of the two modal features of the transformer calculated by multi-head self-attention, the fused feature information with the position label can be obtained, so that the spectral sequence problem between a large number of spectral bands in HSI images can be better solved, especially the long-distance spectral sequence problem. In order to validate the effectiveness of the proposed MVSTF-Net method, three publicly available HSI and LiDAR datasets were selected for comparison and ablation experiments. The overall classification accuracy was 94.12%, 94.26%, and 99.17%, respectively.