Skip to main content
Springer Nature Link
Log in

Assessing the Uncertainty of the Text Generating Process Using Topic Models

  • Conference paper
  • First Online:
ECML PKDD 2020 Workshops (ECML PKDD 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1323))

Abstract

Latent Dirichlet Allocation (LDA) is one of the most popular topic models employed for the analysis of large text data. When applied repeatedly to the same text corpus, LDA leads to different results. To address this issue, several methods have been proposed. In this paper, instead of dealing with this methodological source of algorithmic uncertainty, we assess the aleatoric uncertainty of the text generating process itself. For this task, we use a direct LDA-model approach to quantify the uncertainty due to the random process of text generation and propose three different bootstrap approaches to resample texts. These allow to construct uncertainty intervals of topic proportions for single texts as well as for text corpora over time. We discuss the differences of the uncertainty intervals derived from the three bootstrap approaches and the direct approach for single texts and for aggregations of texts. We present the results of an application of the proposed methods to an example corpus consisting of all published articles in a German daily quality newspaper of one full year and investigate the effect of different sample sizes to the uncertainty intervals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Agrawal, A., Fu, W., Menzies, T.: What is wrong with topic modeling? And how to fix it using search-based software engineering. Inf. Softw. Technol. 98, 74–88 (2018). https://doi.org/10.1016/j.infsof.2018.02.005

    Article  Google Scholar 

  2. Benoit, K., Laver, M., Mikhaylov, S.: Treating words as data with error: uncertainty in text statements of policy positions. Am. J. Polit. Sci. 53(2), 495–513 (2009). https://doi.org/10.1111/j.1540-5907.2009.00383.x

    Article  Google Scholar 

  3. Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012). https://doi.org/10.1145/2133806.2133826

    Article  Google Scholar 

  4. Blei, D.M., Lafferty, J.D.: A correlated topic model of science. Ann. Appl. Stat. 1(1), 17–35 (2007). https://doi.org/10.1214/07-AOAS114

    Article  MathSciNet  MATH  Google Scholar 

  5. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003). https://doi.org/10.1162/jmlr.2003.3.4-5.993

    Article  MATH  Google Scholar 

  6. Chang, J., Boyd-Graber, J., Gerrish, S., Wang, C., Blei, D.M.: Reading tea leaves: how humans interpret topic models. In: Proceedings of the 22nd International NIPS-Conference, pp. 288–296. Curran Associates Inc. (2009). https://dl.acm.org/doi/10.5555/2984093.2984126

  7. Efron, B.: Bootstrap methods: another look at the jackknife. Ann. Stat. 7(1), 1–26 (1979), http://www.jstor.org/stable/10.2307/2958830

  8. Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman & Hall/CRC, New York (1994). https://doi.org/10.1201/9780429246593

  9. Feinerer, I., Hornik, K., Meyer, D.: Text mining infrastructure in R. J. Stat. Softw. 25(5), 1–54 (2008). https://doi.org/10.18637/jss.v025.i05

    Article  Google Scholar 

  10. Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Natl. Acad. Sci. 101(suppl 1), 5228–5235 (2004). https://doi.org/10.1073/pnas.0307752101

    Article  Google Scholar 

  11. Koppers, L., Rieger, J., Boczek, K., von Nordheim, G.: tosca: Tools for Statistical Content Analysis (2019). https://doi.org/10.5281/zenodo.3591068, R package version 0.1-5

  12. Krippendorff, K.: Content Analysis: An Introduction to Its Methodology, 3rd edn. Sage Publications, Thousand Oaks (2013)

    Google Scholar 

  13. Maier, D., et al.: Applying LDA topic modeling in communication research: toward a valid and reliable methodology. Commun. Methods Measures 12(2–3), 93–118 (2018). https://doi.org/10.1080/19312458.2018.1430754

    Article  Google Scholar 

  14. Mimno, D., Wallach, H.M., Talley, E., Leenders, M., McCallum, A.: Optimizing semantic coherence in topic models. In: Proceedings of the 2011 EMNLP-Conference, pp. 262–272. ACL (2011). https://dl.acm.org/doi/10.5555/2145432.2145462

  15. Newman, D., Bonilla, E.V., Buntine, W.: Improving topic coherence with regularized topic models. In: Proceedings of the 24th International NIPS-Conference, pp. 496–504. Curran Associates Inc. (2011). https://dl.acm.org/doi/10.5555/2986459.2986515

  16. Nguyen, V.A., Boyd-Graber, J., Resnik, P.: Sometimes average is best: the importance of averaging for prediction using MCMC inference in topic modeling. In: Proceedings of the 2014 EMNLP-Conference, pp. 1752–1757. ACL (2014). https://doi.org/10.3115/v1/D14-1182

  17. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2019). http://www.R-project.org/

  18. Rieger, J.: ldaPrototype: a method in R to get a prototype of multiple latent Dirichlet allocations. J. Open Source Softw. 5(51), 2181 (2020). https://doi.org/10.21105/joss.02181

    Article  Google Scholar 

  19. Rieger, J., Koppers, L., Jentsch, C., Rahnenführer, J.: Improving Reliability of Latent Dirichlet Allocation by Assessing Its Stability Using Clustering Techniques on Replicated Runs (2020). https://arxiv.org/abs/2003.04980

  20. Rieger, J., Rahnenführer, J., Jentsch, C.: Improving latent Dirichlet allocation: on reliability of the novel method LDA Prototype. In: Métais, E., Meziane, F., Horacek, H., Cimiano, P. (eds.) NLDB 2020. LNCS, vol. 12089, pp. 118–125. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-51310-8_11

    Chapter  Google Scholar 

  21. Roberts, M.E., Stewart, B.M., Tingley, D., Airoldi, E.M.: The structural topic model and applied social science. In: NIPS-Workshop on Topic Models: Computation, Application, and Evaluation (2013)

    Google Scholar 

  22. Rosen-Zvi, M., Griffiths, T., Steyvers, M., Smyth, P.: The author-topic model for authors and documents. In: Proceedings of the 20th UAI-Conference, pp. 487–494. AUAI Press (2004). https://dl.acm.org/doi/10.5555/1036843.1036902

  23. Stevens, K., Kegelmeyer, P., Andrzejewski, D., Buttler, D.: Exploring topic coherence over many models and many topics. In: Proceedings of the 2012 Joint EMNLP/CoNLL-Conference, pp. 952–961. ACL (2012). https://dl.acm.org/doi/10.5555/2390948.2391052

  24. Wald, A.: Tests of statistical hypotheses concerning several parameters when the number of observations is large. Trans. Am. Math. Soc. 54, 426–482 (1943). https://doi.org/10.1090/S0002-9947-1943-0012401-3

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The present study is part of a project of the Dortmund Center for data-based Media Analysis (DoCMA). In addition, the authors gratefully acknowledge the computing time provided on the Linux HPC cluster at TU Dortmund University (LiDO3), partially funded in the course of the Large-Scale Equipment Initiative by the German Research Foundation (DFG) as project 271512359.

Author information

Authors and Affiliations

  1. Department of Statistics, TU Dortmund University, 44221, Dortmund, Germany

    Jonas Rieger, Carsten Jentsch & Jörg Rahnenführer

Authors
  1. Jonas Rieger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carsten Jentsch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jörg Rahnenführer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jonas Rieger .

Editor information

Editors and Affiliations

  1. University of Sydney, Sydney, NSW, Australia

    Irena Koprinska

  2. Monash University, Clayton, VIC, Australia

    Michael Kamp

  3. University of Bari Aldo Moro, Bari, Italy

    Annalisa Appice

  4. University of Bari Aldo Moro, Bari, Italy

    Corrado Loglisci

  5. University of Guelph, Guelph, ON, Canada

    Luiza Antonie

  6. University of Caen Normandy, Caen, France

    Albrecht Zimmermann

  7. University of Pisa, Pisa, Italy

    Riccardo Guidotti

  8. Norwegian University of Science and Technology, Trondheim, Norway

    Özlem Özgöbek

  9. University of Porto, Porto, Portugal

    Rita P. Ribeiro

  10. UPC BarcelonaTech, Barcelona, Spain

    Ricard Gavaldà

  11. University of Porto, Porto, Portugal

    João Gama

  12. Fraunhofer IAIS, St. Augustin, Germany

    Linara Adilova

  13. Royal Holloway University of London, Egham, UK

    Yamuna Krishnamurthy

  14. University of Lisbon, Lisbon, Portugal

    Pedro M. Ferreira

  15. University of Bari Aldo Moro, Bari, Italy

    Donato Malerba

  16. University of Lisbon, Lisbon, Portugal

    Ibéria Medeiros

  17. University of Bari Aldo Moro, Bari, Italy

    Michelangelo Ceci

  18. ICAR-CNR, Rende, Italy

    Giuseppe Manco

  19. University of Naples Federico II, Naples, Italy

    Elio Masciari

  20. University of North Carolina, Charlotte, NC, USA

    Zbigniew W. Ras

  21. Australian National University, Canberra, ACT, Australia

    Peter Christen

  22. Leibniz University Hannover, Hannover, Germany

    Eirini Ntoutsi

  23. Technical University of Dortmund, Dortmund, Germany

    Erich Schubert

  24. University of Southern Denmark, Odense, Denmark

    Arthur Zimek

  25. University of Pisa, Pisa, Italy

    Anna Monreale

  26. Warsaw University of Technology, Warsaw, Poland

    Przemyslaw Biecek

  27. ISTI-CNR, PISA, Italy

    Salvatore Rinzivillo

  28. Berlin Institute of Technology, Berlin, Germany

    Benjamin Kille

  29. Berlin Institute of Technology, Berlin, Germany

    Andreas Lommatzsch

  30. Norwegian University of Science and Technology, Trondheim, Norway

    Jon Atle Gulla

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rieger, J., Jentsch, C., Rahnenführer, J. (2020). Assessing the Uncertainty of the Text Generating Process Using Topic Models. In: Koprinska, I., et al. ECML PKDD 2020 Workshops. ECML PKDD 2020. Communications in Computer and Information Science, vol 1323. Springer, Cham. https://doi.org/10.1007/978-3-030-65965-3_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-65965-3_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-65964-6

  • Online ISBN: 978-3-030-65965-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships