Answers are grounded in this paper and cite their sources.
Assesses topic modeling improvements using embeddings from pre-trained language models, indicating enhanced integration and accuracy in modeling.
Topic modeling is traditionally applied to word counts without accounting for the context in which words appear. Recent advances in large language models (LLMs) offer contextualized word embeddings that capture deeper semantic meaning and relationships. We aim to leverage such embeddings to improve topic modeling.We use a pre-trained LLM to convert each document into a sequence of word embeddings. This sequence is then modeled as a Poisson point process, with its intensity measure expressed as a convex combination of K base measures, each corresponding to a topic. To estimate these topics, we propose a flexible algorithm that integrates traditional topic modeling methods, enhanced by net-rounding applied before and kernel smoothing applied after. One advantage of this framework is that it treats the LLM as a black box, requiring no fine-tuning of its parameters. Another advantage is its ability to seamlessly integrate any traditional topic modeling approach as a plug-in module, without the need for modifications.Assuming each topic is a β-Hölder smooth intensity measure on the embedded space, we establish the rate of convergence of our method. We also provide a minimax lower bound and show that the rate of our method matches with the lower bound when β≤1. Additionally, we apply our method to two real datasets, providing evidence that it offers an advantage over previous topic modeling approaches.
No takes yet. Share an insight, caveat, or question.
Austern et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: