The Geometry of Language: Decoding the Skip-Gram Algorithm

This presentation, The Geometry of Language: Decoding the Skip-Gram Architecture from Neural Anatomy to Semantic Emergence, offers a rigorous yet intuitive exploration of how meaning in language can be transformed into mathematical structure through the Skip-Gram model. Beginning with the foundational linguistic insight that “a word is known by the company it keeps,” the work establishes that meaning does not reside in isolated words but in patterns of contextual co-occurrence. It then contrasts the Continuous Bag-of-Words (CBOW) and Skip-Gram architectures, highlighting the latter’s inversion of the learning objective—predicting the target from context—which enables better handling of rare words through pairwise training. The presentation proceeds with a step-by-step anatomical breakdown of the neural process, including encoding using one-hot vectors, projecting into a hidden embedding space, and generating probabilistic outputs via softmax. A key conceptual shift is emphasised: the true objective is not prediction accuracy but the emergence of a structured embedding space in which semantic relationships are encoded geometrically. This is empirically demonstrated through training dynamics, where probability mass concentrates on true context words, and through visualisations of similarity matrices and vector relationships. The framework is then scaled to a real corpus—Shakespeare—revealing how embeddings capture thematic, narrative, and functional associations across a large vocabulary. Nearest-neighbour analysis and t-SNE projections further illustrate how words cluster based on shared usage rather than explicit rules. The presentation culminates in a unifying insight: language can be understood as a high-dimensional geometric system where co-occurrence patterns shape distances and directions in vector space, making embeddings the bridge between raw text and machine understanding.