When More Metrics Do Not Improve Faithfulness Assessment: A Document-Level Evaluation of AI-Generated News Summaries

Automatic scores are attractive for screening artificial intelligence (AI)-generated news summaries, but their agreement with human judgments must be evaluated without sharing source articles between model fitting and testing. We examined whether a fixed linear fusion of five published scores improves the ranking of human-rated source faithfulness over the published entailment score alone. We reanalysed the Creative Commons licensed XSum hallucination-annotation release, using its existing human judgments and precomputed ROUGE-1, ROUGE-2, ROUGE-L, BERTScore, and entailment scores. Before inspecting outcomes, we fixed mechanical eligibility rules, a deterministic document-level 80/20 split, ridge regression with penalty 1.0, Spearman correlation as the primary endpoint, and 2,000 document-cluster bootstrap draws. The eligible file contained 1,992 system-summary records from 498 BBC article IDs; 399 articles and 1,596 records were used for fitting, and 99 articles and 396 records for testing. On the held-out set, fusion correlated with human faithfulness at ρ = 0.425 (95% bootstrap interval 0.325 to 0.515), versus ρ = 0.445 (0.342 to 0.532) for entailment. The prespecified difference was 0.020 ( 0.100 to 0.066). For the secondary factuality-vote target, correlations were 0.285 and 0.285 after rounding, with no detectable fusion advantage. This reanalysis shows no evidence that this fixed fusion improves ranking over entailment in the released sample. The labels do not assess source attribution or editorial fidelity, and no new human annotation or summary generation was performed.