Masked self-supervised pre-training for steel corrosion recognition via vision transformer under limited sample conditions

The identification of steel corrosion is a critical task in the field of engineering maintenance; however, the high cost of collecting corrosion samples has limited the application of traditional identification algorithms. This challenge is particularly acute for Transformer architecture models, which require a substantial amount of training data and thus face significant difficulties in corrosion identification. Addressing this issue, this study explores a pre-training strategy for Transformer architectures aimed at recognizing the state of steel corrosion. By employing masked self-supervised learning for pre-training the model under conditions of limited training samples and subsequently refining it through fine-tuning, this approach enables precise identification of steel corrosion states. Experimental results on a custom datasets demonstrate that this method, without increasing the cost of additional annotations, effectively utilizes the limited sample information for learning and modeling, thereby significantly enhancing model accuracy, which provides new insights and tools for addressing similar engineering problems and suggests that self-supervised learning could become an important technological trend in engineering maintenance, especially in contexts where sample acquisition costs are high.