SynGhost: Imperceptible and Universal Task-agnostic Backdoor Attack in Pre-trained Language Models

Pre-training has been a necessary phase for deploying pre-trained language models (PLMs) to achieve remarkable performance in downstream tasks. However, we empirically show that backdoor attacks exploit such a phase as a vulnerable entry point for task-agnostic. In this paper, we first propose maxEntropy, an entropy-based poisoning filtering defense, to prove that existing task-agnostic backdoors are easily exposed, due to explicit triggers used. Then, we present SynGhost, an imperceptible and universal task-agnostic backdoor attack in PLMs. Specifically, SynGhost hostilely manipulates clean samples through different syntactic and then maps the backdoor to representation space without disturbing the primitive representation. SynGhost further leverages contrastive learning to achieve universal, which performs a uniform distribution of backdoors in the representation space. In light of the syntactic properties, we also introduce an awareness module to alleviate the interference between different syntactic. Experiments show that SynGhost holds more serious threats. Not only do severe harmfulness to various downstream tasks on two tuning paradigms but also to any PLMs. Meanwhile, SynGhost is imperceptible against three countermeasures based on perplexity, fine-pruning, and the proposed maxEntropy.