Adversarial Example Transfer Method for Vision-Language Pre-Training Models Based on Negative Sample Feature Perturbation

To address the issue of insufficient transferability of existing adversarial example generation methods for vision-language pre-training (VLP) models, this paper proposes an adversarial example transfer method for VLP models based on negative sample feature perturbation. First, a novel cross-modal collaborative perturbation strategy is constructed. By introducing negative samples into the cross-modal perturbation mechanism, the strategy explores more perturbation directions, breaks the original modal alignment constraints and avoids the local focus of adversarial perturbations. Then, to reduce the computational cost, a dynamic threshold attack strategy is built to measure the modal similarity of the generated adversarial examples. Finally, with the help of a multi-modal fusion encoder, a cross-modal fusion semantic attack (CFSA) module is designed. This module extracts the middle-layer features of image-text pairs and improves the transfer attack effect of adversarial examples. The proposed attack method is experimentally evaluated on the Flickr30K and MSCOCO datasets. The results show that for the adversarial examples generated on the Flickr30K dataset, the attack success rate (ASR) of the proposed method reaches up to 95.3% on multiple black-box models; for those generated on the MSCOCO dataset, the maximum attack success rate on multiple black-box models reaches 70.17%. Compared with the current methods, the adversarial examples generated by the proposed method achieve better attack performance.