Answers are grounded in this paper and cite their sources.
Topic modeling is a text analysis technique for automatically discovering common themes in a collection of documents. "Human-in-the-loop" topic modeling (HLTM) allows domain experts to steer and adjust the creation of topic models. In this case study, we use a custom-built HLTM interface to assess the impact of human refinement on model interpretability and predictive performance in collaboration with an analytics team within our organization. Using a small dataset (≈ 12k documents) of responses drawn from an organizational employee satisfaction survey, we compare the pre- and post-refinement models using both human judgments and automated metrics. We find that human refinement can enhance interpretability and predictive performance in some cases, but may lead to overfitting on the training data, which negatively impacts model quality. Furthermore, we observe that existing evaluation methods don't sufficiently and clearly capture topic model quality in applied settings, and propose guidance for further HLTM tool development.
No takes yet. Share an insight, caveat, or question.
Gao et al. (2024) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: