From Word to Sentence Embedding and beyond: Bridging the Gap in Text Representation Survey

Text representation is fundamental to Natural Language Processing (NLP), as it transforms human language into numbers (vectors) that machines can interpret. The quality of the representation directly determines the effectiveness of downstream tasks such as sentiment analysis, translation, and question answering. This survey adopts a multidimensional evaluation framework to analyze text representation techniques, focusing on their evolution, performance, and applications across various NLP tasks. It proposes a text representation taxonomy that categorizes embedding methods based on their granularity for specific use cases. Various evaluation metrics often used in assessing embedding performance were discussed. In addition, the survey reviewed significant data catalogues relative to their applicable NLP task, which is crucial in generating an embedding. It provides a critical analysis of emerging trends, while offering insights into their potential to address scalability and domain adaptation challenges. The paper’s unique contribution lies in its comprehensive projection of historical developments, current advancements, and future directions, providing researchers and practitioners with a simple yet valuable understanding of the field.