CompKG-FactEval: a knowledge graph comparison-based framework for summarization factual consistency evaluation

Factual errors in abstractive summarization undermine its reliability and real-world applicability. Existing factual consistency evaluation methods, mainly those based on pretrained language models (PLMs) and large language models (LLMs), focus on local and linear context while neglecting global document information and structured semantic knowledge. Their reliance on domain-biased data limits generalization, and their black-box nature reduces interpretability. We propose CompKG-FactEval, a framework for factual consistency evaluation via factual knowledge graph comparison. CompKG-FactEval constructs Factual Knowledge Heterogeneous Graphs (FK-Graphs) for both the document and its summary, integrating global factual knowledge with structured linguistic information. It represents graph nodes using pretrained contextual and knowledge embeddings, while an HGMAE-based encoder is additionally examined as an optional extension for incorporating neighborhood-level structural signals. Finally, we design a novel graph kernel, S 2 -K, that jointly models syntactic and semantic similarity to efficiently compare FK-Graphs and assess factual consistency. Beyond improving evaluation accuracy, CompKG-FactEval enhances interpretability by localizing factual inconsistencies within the summary. We evaluate CompKG-FactEval on two datasets—DeFacto, which contains PLM-generated summaries, and UniSumEval, which contains LLM-generated summaries. The results show that CompKG-FactEval outperforms state-of-the-art PLM- and LLM-based methods on DeFacto and achieves the highest correlation with human judgments on UniSumEval across general and domain-specific datasets, demonstrating robust and domain-agnostic performance.