What do you learn from context? Probing for sentence structure in contextualized word representations

Contextualized representation models such as ELMo (Peters et al. , 2018a) and (Devlin et al. , 2018) have recently achieved state-of-the-art results on a array of downstream NLP tasks. Building on recent token-level probing, we introduce a novel edge probing task design and construct a broad suite sub-sentence tasks derived from the traditional structured NLP pipeline. We word-level contextual representations from four recent models and how they encode sentence structure across a range of syntactic, , local, and long-range phenomena. We find that existing models trained language modeling and translation produce strong representations for phenomena, but only offer comparably small improvements on semantic over a non-contextual baseline.