June 29, 2018289 citationsOpen Access
Sparse Additive Generative Models of Text
Key Points
- This research aims to introduce a new generative model for text that enhances sparsity and reduces complexity in inference.
- Proposed a model using log-frequency deviations from a background distribution for class labels.
- Implemented techniques to enforce sparsity and facilitate addition in log space.
- Applied the model across various scenarios including classification and topic modeling.
- The new model exhibits improved performance in classification tasks compared to traditional methods.
- Successfully demonstrates reduced complexity in multifaceted generative models without latent variables.
- Achieves effective topic modeling with better interpretability and efficiency.
Abstract
Generative models of text typically associate a multinomial with every class label or topic. Even in simple models this requires the estimation of thousands of parameters; in multifaceted latent variable models, standard approaches require additional latent ``switching'' variables for every token, complicating inference. In this paper, we propose an alternative generative model for text. The central idea is that each class label or latent topic is endowed with a model of the deviation in log-frequency from a constant background distribution. This approach has two key advantages: we can enforce sparsity to prevent overfitting, and we can combine generative facets through simple addition in log space, avoiding the need for latent switching variables. We demonstrate the applicability of this idea to a range of scenarios: classification, topic modeling, and more complex multifaceted generative models.