CORPURES: Benchmark corpus for urdu extractive summaries and experiments using supervised learning

Text summarization is the process of shortening the text so that it conveys the key points. Several text summarization methods and benchmark corpora are available for languages like English. A significant hurdle in developing and evaluating existing or new text summarization methods is the unavailability of standardized benchmark corpora, especially for South Asian languages. Among other things, a reference corpus enables researchers to compare existing state-of-the-art methods. Our study addresses this gap by developing a benchmark corpus for one of the widely spoken yet under-resourced language Urdu. The reported corpus contains 161 documents with manually written extractive summaries from the newswire domain. We also perform several experiments on the corpus to show how it can be used to develop, evaluate, and compare text summarization systems using a supervised learning approach for the Urdu language. Our results show that the state of the art classifiers are good candidates for Urdu text summarization when supervised learning techniques are employed. Also, a radical word segmentation technique such as fixed-length segmentation outperforms all other settings (Senetnce Match F1=57%, ROUGE-2 F1=64.4%). On the basic preprocessing of Urdu texts, we observe that tokenization of words on space is a reliable approach until the proper word segmentation tools for Urdu are mature enough. On word similarity features needed for supervised learning, it is observed that a radical stemming such as Ultra stemming with length (1 and 2) works better than the existing stemming and lemmatization tools for Urdu. Finally, the artificially generated datasets do not significantly improve results compared to the original data.