In search of semantic distance: metaphoric and non-metaphoric constructions in static and contextual representations

Phrases such as burning question, digital waste or invasion of technology are relatively ordinary expressions understood by any speaker of English. While diverse in structure and meaning, they demonstrate a semantic tension between the basic meanings of a metaphoric and a non-metaphoric constituent. Albeit frequent in discourse, they remain a challenge for automatic language processing systems, especially for smaller, less represented languages. In this work, we inspect a broad array of language models to embed noun phrases in Slovene and investigate the potential of word embeddings to identify metaphoric phrases via the semantic distance of its constituents as measured via cosine similarity. The study shows both static and contextual monolingual embeddings encode relevant semantic information while multilingual embeddings demonstrate no significant effect in this experimental setting. Moreover, the study unravels the most effective layers for basic meaning representation and highlights the influence of other, non-semantic factors on cosine similarity. By shedding light on these mechanisms, the study provides new insights for both metaphor processing and our understanding of the inner workings of language models.