Key points are not available for this paper at this time.
Recent work using auxiliary prediction task classifiers to investigate the of LSTM representations has begun to shed light on why pretrained, like ELMo (Peters et al. , 2018) and CoVe (McCann et al. , 2017), are so beneficial for neural language understanding models. We still, , do not yet have a clear understanding of how the choice of pretraining affects the type of linguistic information that models learn. With in mind, we compare four objectives---language modeling, translation, -thought, and autoencoding---on their ability to induce syntactic and-of-speech information. We make a fair comparison between the tasks by constant the quantity and genre of the training data, as well as the architecture. We find that representations from language models perform best on our syntactic auxiliary prediction tasks, even trained on relatively small amounts of data. These results suggest that modeling may be the best data-rich pretraining task for transfer applications requiring syntactic information. We also find that the from randomly-initialized, frozen LSTMs perform strikingly well our syntactic auxiliary tasks, but this effect disappears when the amount of data for the auxiliary tasks is reduced.
Zhang et al. (Wed,) studied this question.