The main intrinsic evaluation for vector space representation has been focused on textual similarity, where the task is to predict how semantically similar two words or sentences are. We propose a novel framework, Story Cloze Evaluator, for evaluating vector representations which goes beyond textual similarity and captures the notion of predicting what should happen next given a context. This evaluation methodology is simple to run, scalable, reproducible by the community, non-subjective, 100% agreeable by human, and challenging to the state-of-theart models, which makes it a promising new framework for further investment of the representation learning community.
No takes yet. Share an insight, caveat, or question.
Mostafazadeh et al. (2016) studied this question.