Robust Fine‐Tuning Based on Fuzz Testing for Chinese BERT‐Based PLMs

With the development of Pre‐trained Language Models (PLMs), a wide variety of Chinese Natural Language Processing (NLP) tasks benefit from the Bidirectional Encoder Representations of Transformers (BERT) structure. Unfortunately, Chinese BERT‐based PLMs are vulnerable to adversarial attacks, exposing robustness issues. It inspired numerous defence works devoted to improving the model robustness. However, existing pre‐training methods incur substantial time costs, while fine‐tuning methods need more evidence to demonstrate how well the target model has been defended. This work aims to bridge this gap. We propose RoFuBERT, a robust fine‐tuning framework using fuzz testing for Chinese BERT‐based PLMs. We extract Chinese basis, glyph, pinyin and semantic features of downstream NLP tasks during fine‐tuning. We integrate them into fuzz testing and evaluate the testing completeness of the model under adversarial attacks. Finally, we retrain the model for robust fine‐tuning. Our evaluation shows that, compared with the two baselines, RoFuBERT efficiently improves adversarial robustness of models. RoFuBERT takes only 0.69 times more time, and the reduction in adversarial attack success rate is improved by 37.76%. The modification rate of the adversarial samples that can deceive models is improved by 5.19%, on average.