Answers are grounded in this paper and cite their sources.
We develop an asymptotic theory for the partition function of the word embeddings model word2vec. The proof involves a study of properties of matrices, their determinants and distributions of random normal vectors when their dimension tends to infinity. The conditions imposed are mild enough to cover practically important situations. The implication is that for any word i from a vocabulary W, the context vector ci is a reflection of the word vector wi in approximately half of the dimensions. This allows us to halve the number of trainable parameters in static word embedding models.
No takes yet. Share an insight, caveat, or question.
Mynbaev et al. (2022) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: