Contextualized embeddings for semantic change detection: Lessons learned

This paper presents a qualitative analysis of the outputs of contextualized embedding-based methods for detecting diachronic semantic change. We also introduce an ensemble-method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can sometimes predict high change scores for words which are not undergoing any real diachronic semantic shift. Such cases are discussed in detail with examples, and their linguistic categorization is proposed. Contextualized embeddings are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stems from their distributional nature. Additionally, they often merge together syntactic and semantic aspects of lexical entities. Finally, we propose possible solutions to these issues.