Dynamics of Masked Image Modeling in Hyperspectral Image Classification

Masked image modeling (MIM), a common selfsupervised learning (SSL) technique, has been extensively studied for remote sensing image processing. Nevertheless, its effectiveness for hyperspectral imagery (HSI) remains underexplored due to the distinct data structures and high dimensionality. This paper aims to provide a detailed understanding of MIM from different perspectives of representation learning and statistical analysis for HSI classification tasks. Our study reveals that the MIM paradigm injects inductive bias in the attention mechanism of the transformer model, which is advantageous for capturing the local discrepancies between the spectra. We also show that MIM can increase the diversity of the attention heads in every layer, which is beneficial for the model in extracting more discriminative features from different spectral bands. The similarity of representations from various layers further proves this. Furthermore, our investigation highlights how MIM introduces a dynamic perspective to spectral representations, enabling the model to learn more robust and discriminative features. The final numerical experiments indicate that a moderate mask ratio can enhance the performance of downstream tasks. This suggests that designing a more targeted masking strategy might be necessary to achieve higher and more stable gains in downstream task performance. Without bells and whistles, the vanilla MIM improves the overall classification accuracy by an average of 2.69% over its SL counterpart. We hope that our findings can advance the understanding of MIM in HSI and inspire the design of a more stable SSL paradigm for HSI processing.