Evaluation of sentence embeddings in downstream and linguistic probing tasks

Despite the fast developmental pace of new sentence embedding methods, it is challenging to find comprehensive evaluations of these different. In the past years, we saw significant improvements in the field of embeddings and especially towards the development of universal encoders that could provide inductive transfer to a wide variety of tasks. In this work, we perform a comprehensive evaluation of recent using a wide variety of downstream and linguistic feature probing. We show that a simple approach using bag-of-words with a recently language model for deep context-dependent word embeddings proved to better results in many tasks when compared to sentence encoders trained entailment datasets. We also show, however, that we are still far away from universal encoder that can perform consistently across several downstream.