Word Embedding: The Geometry of Meaning

This presentation provides a comprehensive introduction to how modern natural language processing (NLP) systems transform text into meaningful mathematical representations. It begins by examining traditional approaches such as one-hot encoding, highlighting their limitations, including high dimensionality, sparsity, and the inability to capture semantic relationships. The concept of orthogonality is introduced to explain why such representations fail to encode meaning effectively. The presentation then introduces TF-IDF as an improved statistical method that balances term frequency and document rarity to produce weighted word representations. Through structured examples, it demonstrates how TF-IDF enables document comparison and similarity measurement, while also acknowledging its limitations in capturing deeper semantic structures. Building on these foundations, the presentation transitions to word embeddings, where meaning is represented as vectors in a continuous geometric space. It explains how semantic relationships can be modelled through vector arithmetic, illustrated by analogies such as “King − Man + Woman = Queen.” The concept of directional transformations is explored, emphasising that semantic shifts are consistent and interpretable within a structured embedding space. Cosine similarity is presented as the central mechanism for evaluating semantic alignment, focusing on directional similarity rather than magnitude. The presentation highlights how cosine similarity enables analogy resolution and supports semantic reasoning by identifying the closest matching vectors. Finally, the presentation explores higher-level geometric interpretations, including semantic clustering, manifold structures, and outlier detection. These concepts demonstrate how embeddings can organise words into meaningful groups and reveal underlying linguistic patterns. Overall, the work provides a clear, visual, and mathematical framework for understanding how machines interpret language through geometry and vector-based representations.