TOPICScore: Evaluating Automatic Text Summarization Using Embeddings, Occurrences, and Topic Detection

Most of the studies published on Text Summarization are evaluated using ROUGE as a metric. Despite its popular-ity for Extractive Methods, its inability to capture semantic prop-erties makes it less suitable for Abstractive Methods. Our work presents a new method for evaluating Text Summarization by creating a trade-off between Extractiveness and Abstractiveness. This is done by using Embeddings, Word Occurrences, and Topic Detection in order to compute a global score named TOPICScore. This score is designed according to three aspects: Compression, Overlapping, and Semantic Similarity. By incorporating these three dimensions, TOPICScore is more agnostic to the used approach. We showed how our score can be computed using Arabic Headline Summary (AHS) dataset and a fine-tuned AraBART model.