Abstract
Pre-trained models with latent variables have been proved to be an effective method in the diverse dialogue generation. However, the latent variables in current models are finite and uninformative, making the generated responses lack diversity and informativeness. In order to address this problem, we propose an exemplar guided latent pre-trained dialogue generation model to sample the latent variables from a continuous sentence embedding space, which can be controlled by the exemplar sentences. The proposed model contains two parts: exemplar seeking and response generation. First, the exemplar seeking builds a sentence graph based on the given dataset and seeks an enlightened exemplar from the graph. Next, the response generation constructs informative latent variables based on the exemplar and generates diverse responses with latent variables. Experiments show that the model can effectively improve the propriety and diversity of responses and achieve state-of-the-art performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Dong, L., et al.: Unified language model pre-training for natural language understanding and generation. In: NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada (2019)
Sun, Y., et al.: ERNIE 2.0: a continual pre-training framework for language understanding. In: AAAI, 7–12 February 2020, New York, NY, USA (2020)
Xiao, D., et al.: ERNIE-GEN: an enhanced multi-flow pre-training and fine-tuning framework for natural language generation. In: Proceedings of IJCAI 2020 (2020)
Rashkin, H., Smith, E.M., Li, M., Boureau, Y.-L.: Towards empathetic open-domain conversation models: a new benchmark and dataset. In: Proceedings of ACL 2019, 28 July–2 August 2019, Florence, Italy (2019)
Zhang, Y., et al.: DIALOGPT: large-scale generative pre-training for conversational response generation. In: Proceedings of ACL 2020, Online, 5–10 July 2020 (2020)
Zeng, Y., Nie, J.: Generalized conditioned dialogue generation based on pre-trained language model. CoRR, vol. abs/2010.11140 (2020)
Zhao, X., Wu, W., Xu, C., Tao, C., Zhao, D., Yan, R.: Knowledge-grounded dialogue generation with pre-trained language models. In: Proceedings of EMNLP 2020, Online, 16–20 November 2020 (2020)
Yang, Z., et al.: StyleDGPT: stylized response generation with pre-trained language models. In: Proceedings of Findings, EMNLP 2020, Online Event, 16–20 November 2020 (2020)
Wu, C.-S., Hoi, S.C., Socher, R., Xiong, C.: TOD-BERT: pre-trained natural language understanding for task-oriented dialogue. In: Proceedings of EMNLP, Online (2020)
Zheng, Y., Zhang, R., Huang, M., Mao, X.: A pre-training based personalized dialogue generation model with persona-sparse data. In: Proceedings of AAAI 2020, 7–12 February 2020, New York, NY, USA (2020)
Cao, Y., Bi, W., Fang, M., Tao, D.: Pretrained language models for dialogue generation with multiple input sources. In: Proceedings of EMNLP 2020, Online Event, 16–20 November 2020 (2020)
Le, H., Hoi, S.C.H.: Video-grounded dialogues with pretrained generation language models. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of ACL 2020, Online, 5–10 July 2020 (2020)
Bao, S., He, H., Wang, F., Wu, H., Wang, H.: PLATO: pre-trained dialogue generation model with discrete latent variable. In: Proceedings of ACL 2020, Online, 5–10 July 2020 (2020)
Bao, S., et al.: PLATO-2: towards building an open-domain chatbot via curriculum learning. CoRR, vol. abs/2006.16779 (2020)
Zhang, L., Yang, Y., Zhou, J., Chen, C., He, L.: Retrieval-polished response generation for chatbot. IEEE Access 8, 123882–123890 (2020)
Shalyminov, I., Sordoni, A., Atkinson, A., Schulz, H.: Hybrid generative-retrieval transformers for dialogue domain adaptation. CoRR, vol. abs/2003.01680 (2020)
Gupta, P., Bigham, J.P., Tsvetkov, Y., Pavel, A.: Controlling dialogue generation with semantic exemplars. CoRR, vol. abs/2008.09075 (2020)
Ma, T., Yang, H., Tian, Q., Tian, Y., Al-Nabhan, N.: A hybrid Chinese conversation model based on retrieval and generation. Future Gener. Comput. Syst. 114, 481–490 (2021)
Weston, J., Dinan, E., Miller, A.H.: Retrieve and refine: improved sequence generation models for dialogue. In: Proceedings of SCAI@EMNLP 2018, 31 October 2018, Brussels, Belgium (2018)
Pandey, G., Contractor, D., Kumar, V., Joshi, S.: Exemplar encoder-decoder for neural conversation generation. In: Proceedings of ACL 2018, Melbourne, Australia, 15–20 July 2018, Volume 1: Long Papers (2018)
Song, Y., Li, C., Nie, J., Zhang, M., Zhao, D., Yan, R.: An ensemble of retrieval-based and generation-based human-computer conversation systems. In: Proceedings of IJCAI 2018, 13–19 July 2018, Stockholm, Sweden (2018)
Yang, L., et al.: A hybrid retrieval-generation neural conversation model. In: Proceedings of CIKM 2019, 3–7 November 2019, Beijing, China (2019)
Zhang, J., Tao, C., Xu, Z., Xie, Q., Chen, W., Yan, R.: EnsembleGAN: adversarial learning for retrieval-generation ensemble model on short-text conversation. In: Proceedings of SIGIR 2019, 21–25 July 2019, Paris, France (2019)
Zhu, Q., Cui, L., Zhang, W., Wei, F., Liu, T.: Retrieval-enhanced adversarial training for neural response generation. In: Proceedings of ACL 2019, 28 July–2 August 2019, Florence, Italy (2019)
Wu, Y., Wei, F., Huang, S., Wang, Y., Li, Z., Zhou, M.: Response generation by context-aware prototype editing. In: Proceedings of AAAI 2019, 27 January–1 February 2019, Honolulu, Hawaii, USA (2019)
Cai, Y.D., et al.: Skeleton-to-response: dialogue generation guided by retrieval memory. In: Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, 2–7 June 2019, Volume 1 (Long and Short Papers) (2019)
Cai, D., Wang, Y., Bi, W., Tu, Z., Liu, X., Shi, S.: Retrieval-guided dialogue response generation via a matching-to-generation framework. In: Proceedings of EMNLP-IJCNLP 2019, 3–7 November 2019, Hong Kong, China (2019)
Cai, H., Chen, H., Song, Y., Zhao, X., Yin, D.: Exemplar guided neural dialogue generation. In: Proceedings of IJCAI 2020 (2020)
Wu, S., Li, Y., Zhang, D., Zhou, Y., Wu, Z.: TopicKA: generating commonsense knowledge-aware dialogue responses towards the recommended topic fact. In: Proceedings of IJCAI 2020 (2020)
Zhang, H., Liu, Z., Xiong, C., Liu, Z.: Grounded conversation generation as guided traverses in commonsense knowledge graphs. In: Proceedings of ACL 2020, Online, 5–10 July 2020 (2020)
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, 2–7 June 2019, Volume 1 (Long and Short Papers) (2019)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: Bengio, Y., LeCun, Y. (eds.) 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14–16 April 2014, Conference Track Proceedings (2014). http://arxiv.org/abs/1312.6114
Zhao, T., Zhao, R., Eskénazi, M.: Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In: Proceedings of ACL 2017, Vancouver, Canada, 30 July–4 August 2017, Volume 1: Long Papers (2017)
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing dialogue agents: i have a dog, do you have pets too? In: Proceedings of ACL 2018, Melbourne, Australia, 15–20 July 2018, Volume 1: Long Papers (2018)
Li, Y., Su, H., Shen, X., Li, W., Cao, Z., Niu, S.: DailyDialog: a manually labelled multi-turn dialogue dataset In: Proceedings of IJCNLP 2017, Taipei, Taiwan, 27 November–1 December 2017 - Volume 1: Long Papers (2017)
AlAmri, H., et al.: Audio visual scene-aware dialog. In: CVPR 2019, 16–20 June 2019, Long Beach, CA, USA (2019)
Golovanov, S., Kurbanov, R., Nikolenko, S.I., Truskovskyi, K., Tselousov, A., Wolf, T.: Large-scale transfer learning for natural language generation. In: Proceedings of ACL 2019, Florence, Italy, 28 July–2 August 2019, Volume 1: Long Papers (2019)
Dinan, E., et al.: The second conversational intelligence challenge (ConvAI2). CoRR, vol. abs/1902.00098 (2019)
Fang, L., Li, C., Gao, J. Dong, W. Chen, C.: Implicit deep latent variable models for text generation. In: Proceedings of EMNLP-IJCNLP 2019, 3–7 November 2019, Hong Kong, China (2019)
Ramon Sanabria, S.P., Metze, F.: CMU sinbads submission for the DSTC7 AVSD challenge. In: AAAI Dialog System Technology Challenge Workshop (2019)
Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. In: Proceedings of NAACL HLT 2016, 12–17 June 2016, San Diego California, USA (2016)
Chen, X., et al.: Microsoft COCO captions: Data collection and evaluation server. CoRR, vol. abs/1504.00325 (2015)
Acknowledgment
This work was supported by National Natural Science Foundation of China (No. 61906187, No. 61976207, No. 61902394).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, M., Fu, P., Lin, Z., Wang, W., Zang, W. (2021). Exemplar Guided Latent Pre-trained Dialogue Generation. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds) Computational Science – ICCS 2021. ICCS 2021. Lecture Notes in Computer Science(), vol 12743. Springer, Cham. https://doi.org/10.1007/978-3-030-77964-1_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-77964-1_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77963-4
Online ISBN: 978-3-030-77964-1
eBook Packages: Computer ScienceComputer Science (R0)