Automatic Text Evaluation through the Lens of Wasserstein Barycenters

A new metric to evaluate text generation based on deep embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric motivated by a new framework relying on optimal transport tools, i.e., distance and barycenter. By modelling the layer output of deep embeddings as a probability distribution rather than by a vector; this framework provides a natural way to aggregate the different through the Wasserstein space topology. In addition, it provides grounds to our metric and offers an alternative to available e.g., MoverScore and BertScore). Numerical evaluation is performed on different tasks: machine translation, summarization, data2text generation image captioning. Our results show that outperforms BERT based metrics and exhibits more consistent behaviour in particular text summarization.