Low Resource Summarization using Pre-trained Language Models

With the advent of Deep Learning-based Artificial Neural Network models, Natural Language Processing (NLP) has witnessed significant improvements in textual data processing in terms of its efficiency and accuracy. However, the research is mostly restricted to high-resource languages such as English, and low-resource languages still suffer from a lack of available resources in terms of training datasets as well as models with even baseline evaluation results. Considering the limited availability of resources for low-resource languages, we propose a methodology for adapting self-attentive transformer-based architecture models (mBERT, mT5) for low-resource summarization, supplemented by the construction of a new baseline dataset (76.5k article, summary pairs) in a low-resource language, Urdu. Choosing news (a publicly available source) as the application domain has the potential to make the proposed methodology useful for reproducing in other languages with limited resources. Our adapted summarization model urT5 with up to 44.78% reduction in size as compared to mT5 can capture contextual information of the low-resource language effectively with an evaluation score (up to 46.35 ROUGE-1, 77 BERTScore) on par with state-of-the-art models in the high-resource language of English (PEGASUS: 47.21, BART: 45.14 on XSUM Dataset) . The proposed method provided a baseline approach toward extractive as well as abstractive summarization with competitive evaluation results in a limited resource setup.