Dynamic Structured Neural Topic Model with Self-Attention Mechanism

This study presents a dynamic structured neural topic model, which can handle the time-series development of topics while capturing their dependencies.Our model captures the topic branching and merging processes by modeling topic dependencies based on a self-attention mechanism.Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly.Our model outperforms a prior dynamic embedded topic model regarding perplexity and coherence, while maintaining sufficient diversity across topics.Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.