WFSS: weighted fusion of spectral transformer and spatial self-attention for robust hyperspectral image classification against adversarial attacks

Abstract The emergence of adversarial examples poses a significant challenge to hyperspectral image (HSI) classification, as they can attack deep neural network-based models. Recent adversarial defense research tends to establish global connections of spatial pixels to resist adversarial attacks. However, it cannot yield satisfactory results when only spatial pixel information is used. Starting from the premise that the spectral band is equally important for HSI classification, this paper explores the impact of spectral information on model robustness. We aim to discover potential relationships between different spectral bands and establish global connections to resist adversarial attacks. We design a spectral transformer based on the transformer structure to model long-distance dependency relationships among spectral bands. Additionally, we use a self-attention mechanism in the spatial domain to develop global relationships among spatial pixels. Based on the above framework, we further explore the influence of both spectral and spatial domains on the robustness of the model against adversarial attacks. Specifically, a weighted fusion of spectral transformer and spatial self-attention (WFSS) is designed to achieve the multi-scale fusion of spectral and spatial connections, which further improves the model’s robustness. Comprehensive experiments on three benchmarks show that the WFSS framework has superior defensive capabilities compared to state-of-the-art HSI classification methods.