Term weighting constitutes one of the foundational pillars of modern information retrieval systems, determining how search engines and text processing applications assess document relevance to user queries. This comprehensive literature review traces the theoretical evolution and practical implications of term weighting methodologies, from the seminal Term Frequency-Inverse Document Frequency (TF-IDF) formulation to the probabilistically-grounded Best Matching 25 (BM25) algorithm. Beginning with the vector space model pioneered by Gerard Salton and the SMART system at Cornell University in the 1960s, this review examines how Karen Spärck Jones's conceptualization of inverse document frequency transformed statistical approaches to term specificity. The subsequent development of the Probabilistic Relevance Framework by Stephen Robertson and colleagues provided the theoretical foundation for BM25, which introduced critical innovations including term frequency saturation and document length normalization. This review synthesizes empirical evidence demonstrating BM25's superior performance across diverse retrieval tasks, examines its widespread adoption in production systems including Elasticsearch and Apache Lucene, and analyzes emerging hybrid architectures that combine lexical and neural retrieval methods. The analysis reveals that despite advances in dense neural retrieval, BM25 remains an indispensable component of modern search infrastructure, particularly for exact match queries and domain-specific applications.
Frederick Kobla Mensah (Sat,) studied this question.