Neural Dynamic Focused Topic Model

Topic models and all their variants analyse text by learning meaningful representations through word co-occurrences. As pointed out by Williamson et al. (2010), such models implicitly assume that the probability of a topic to be active and its proportion within each document are positively correlated. This correlation can be strongly detrimental in the case of documents created over time, simply because recent documents are likely better described by new and hence rare topics. In this work we leverage recent advances in neural variational inference and present an alternative neural approach to the dynamic Focused Topic Model. Indeed, we develop a neural model for topic evolution which exploits sequences of Bernoulli random variables in order to track the appearances of topics, thereby decoupling their activities from their proportions. We evaluate our model on three different datasets (the UN general debates, the collection of NEURIPS papers, and the ACL Anthology dataset) and show that it (i) outperforms state-of-the-art topic models in generalization tasks and (ii) performs comparably to them on prediction tasks, while employing roughly the same number of parameters, and converging about two times faster.

  • Published in:
    AAAI Conference on Artificial Intelligence
  • Type:
    Inproceedings
  • Authors:
    Cvejoski, Kostadin; Sanchez, Ramses; Ojeda, César
  • Year:
    2023

Citation information

Cvejoski, Kostadin; Sanchez, Ramses; Ojeda, César: Neural Dynamic Focused Topic Model, AAAI Conference on Artificial Intelligence, 2023, https://ojs.aaai.org/index.php/AAAI/article/view/26496, Cvejoski.etal.2023a,

Associated Lamarr Researchers

LAMARR Person Sanchez Ramses - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)

Dr. Ramsés Sánchez

Scientific Coordinator Hybrid ML to the profile