Privacy-Preserving Text Summarization through Differentially Private Federated Learning with Transformer Models

Abstract As the need for safe and effective information processing grows, text summarization has become a crucial Natural Language Processing (NLP) application field. In this paper, the new Transformer based privacy-preserving summarisation system is presented. It combines popular transformer models, such as Sequence-to-Sequence, Text-to-Text Transfer Transformer (T5), Bidirectional and Auto-Regressive Transformers (BART), Multilingual BART, and Pre-training with Extracted Gap sentences for Abstractive Summarisation (Pegasus), in a differentially private federated learning setting. Through the architecture, privacy risks are reduced, and collaborative model training is made possible by guaranteeing that raw text data stays localized among decentralized clients. Extensive experiments are carried out on the CNN/DailyMail and XSum datasets, using both extractive and abstractive summarisation paradigms. The findings show that Pegasus regularly outperforms other assessment measures (BLEU-4 at 32.0 and ROUGE-L at 44.1) outperforming other models by 14.43% and 11.78% on average, compared to the BART and mBART models in a federated environment. Performance patterns throughout communication cycles demonstrate the effectiveness of the FedAvg aggregation strategy, as the quality of the model gradually improves. This study compares the computational efficiency, generalizability, and complexity of the model under circumstances that preserve privacy. The results aid in the creation of intelligent summarisation systems that strike a compromise between language quality, scalability, and privacy; these systems may find use in cross-border information systems, healthcare, and the legal sector.