SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

In the summarization domain, a key requirement for summaries is to be consistent with the input document. Previous work has found that language inference (NLI) models do not perform competitively when to inconsistency detection. In this work, we revisit the use of NLI for detection, finding that past work suffered from a mismatch in granularity between NLI datasets (sentence-level), and inconsistency (document level). We provide a highly effective and light-weight called SummaCConv that enables NLI models to be successfully used for task by segmenting documents into sentence units and aggregating scores pairs of sentences. On our newly introduced benchmark called SummaC (Summary Consistency) consisting of six large inconsistency detection datasets, obtains state-of-the-art results with a balanced accuracy of 74. 4%, 5% point improvement compared to prior work. We make the models and datasets: https: //github. com/tingofurro/summac