Automatic Evaluation of Information Ordering: Kendall's Tau

This article considers the automatic evaluation of information ordering, a task underlying many text-based applications such as concept-to-text generation and multidocument summarization. We propose an evaluation method based on Kendall's τ, a metric of rank correlation. The method is inexpensive, robust, and representation independent. We show that Kendall's τ correlates reliably with human ratings and reading times.