PFDP: Privacy-preserving Federated Distillation Method for Pretraining Language Models

Abstract Advances in Natural Language Processing (NLP) have been significantly driven by the adoption of comprehensive pretrained language models (PLMs) such as BERT, RoBERTa. Nevertheless, increasing concerns regarding data privacy and the enforcement of stringent data protection regulations, such as PDPA and GDPR, have highlighted the limitations of traditional centralized machine learning methods. Federated Learning (FL) emerges as a promising solution that mitigates privacy concerns by training models on client devices and aggregating the parameters on a central server, thus avoiding direct data transfer. Despite its considerable potential, FL?s application in NLP faces numerous challenges. These include the need to replicate architectural designs across different devices, the management of non-Independent and Identically Distributed (non-IID) data, and the significant communication overhead caused by frequent transfers of model parameters. Federated Distillation (FD) has been proposed to overcome these challenges by facilitating information transfer through unlabeled public proxy datasets. This approach helps to reduce communication costs and promotes collaboration among different model architectures. However, FD carries potential privacy risks and may result in substantial loss of previously acquired knowledge, thus diminishing the model's effectiveness. To address these issues, we introduce the Privacy-preserving Federated Distillation Method for Pretraining Language Models (PFDP). Our approach distinguishes itself from conventional methods by injecting noise into a select portion of a predetermined dataset, thereby minimizing its impact on the model's utility. Besides, PFDP utilizes transfer learning to improve the generalization abilities of the global model and reduce the impact of catastrophic forgetting. The extensive assessment across many classification tasks illustrates the efficacy of PFDP in enhancing accuracy while safeguarding privacy.