November 28, 2025Open Access

Automatic Dialogue Summarization Evaluation: Combining Human Judgements with ROUGE and BERTScore

Key Points

Summarization quality appears heavily influenced by human evaluations, particularly for conciseness and grammar.
Moderate alignment found between human scoring and automatic metrics like BERTScore.
Analysis of dialogue cohorts using automatic evaluation metrics highlights varying effectiveness of ROUGE and BERTScore.
Findings suggest a need for better practices in dialogue summarization methodologies, focused on human quality measures.

Abstract

Dialogue summarization remains a challenging task due to conversational ambiguity and informality. This work explores the relationship between human evaluation scores and automatic metrics for thirty dialogue summaries, and aims to identify whether these metrics represent human quality perception. Human evaluators rated summaries on accuracy, conciseness, meaning preservation, and grammar using a 5-point scale. Scores were compared with ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore. Results show a moderate alignment between human perception and BERTScore, while ROUGE metrics show weaker correlations. These findings suggest reference metrics capture semantic alignment more effectively than structural quality. Recommendations for improving summarization include incorporating factual consistency and compression-aware training. The study highlights the continued importance of human evaluation in conversational summarization research.

KI fragen

Bookmark

View Full Paper