Continuous Bag-of-Words (CBOW)

This presentation offers a structured and conceptually grounded exploration of the Continuous Bag-of-Words (CBOW) model as a foundational approach to learning word embeddings within the broader evolution of language representation. Beginning with earlier paradigms such as one-hot encoding and TF-IDF, it establishes the limitations of treating words as independent or frequency-based entities and introduces CBOW as a shift toward context-driven learning, where meaning emerges from patterns of co-occurrence. Using a carefully constructed example sentence, the presentation reframes language modelling as a prediction task, demonstrating how surrounding context words are used to infer a target word. It then systematically walks through the forward-pass pipeline, including input encoding, embedding lookup via matrix multiplication, aggregation via averaging, and the generation of output scores, followed by softmax-based probability distributions. The learning process is further clarified through the role of cross-entropy loss and iterative refinement via backpropagation, highlighting how initially random vectors evolve into meaningful geometric structures. A key insight emphasised is that prediction serves merely as a proxy for learning embeddings, which ultimately encode semantic relationships in a continuous vector space. The transition to industry practice is illustrated using a standard library implementation of Word2Vec, reinforcing that abstraction does not alter the underlying geometry or learning principles. The presentation also includes visualisation of the embedding space, revealing semantic clusters, functional groupings, and the central positioning of high-frequency structural words. It concludes by identifying key conceptual anchors, such as the influence of context window size and the limitations of static embeddings in handling polysemy, thereby motivating the need for more advanced models. Overall, the work situates CBOW not only as a computational technique but as a critical step toward modern transformer-based architectures and large language models.