Historical-Domain Pre-trained Language Model for Historical Extractive Text Summarization

In recent years, pre-trained language models (PLMs) have shown remarkable advancements in the extractive summarization task across diverse domains.However, there remains a lack of research specifically in the historical domain.In this paper, we propose a novel method for extractive historical single-document summarization that leverages the potential of a domain-aware historical bidirectional language model, pre-trained on a large-scale historical corpus.Subsequently, we fine-tune the language model specifically for the task of extractive historical single-document summarization.One major challenge for this task is the lack of annotated datasets for historical summarization.To address this issue, we construct a dataset by collecting archived historical documents from the Centre Virtuel de la Connaissance sur l'Europe (CVCE) group at the University of Luxembourg.Furthermore, to better learn the structural features of the input documents, we use a sentence position embedding mechanism that enables the model to learn the position information of sentences.The overall experimental results on our historical dataset collected from the CVCE group show that our method outperforms recent state-of-the-art methods in terms of ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.To the best of our knowledge, this is the first work on extractive historical text summarization.