Skip to main content
Springer Nature Link
Log in

A Semantic Similarity Distance-Aware Contrastive Learning for Abstractive Summarization

  • Conference paper
  • First Online:
PRICAI 2023: Trends in Artificial Intelligence (PRICAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14325))

Included in the following conference series:

Abstract

Recently, contrastive learning has been extended from visual representation to summarization tasks. Abstractive summarization aims to generate a short description for a document while retaining significant information. At present, the methods of contrastive learning summarization focus on modeling the global semantics of source documents, targets and candidate summaries to maximize their similarities. However, they ignore the influence of sentence semantics in the source document. In this paper, we propose a sentence-level semantic similarity distance-aware contrastive learning method (SSDCL), which integrates the semantic similarity distance between summaries and sentences of source documents into the contrastive loss in the form of soft weights. Therefore, our model maximize the similarity between summaries and salient information, while minimizing the similarity between summaries and noise. We conducted extensive experiments on CNN/Daily Mail and XSum datasets to verify our model. The experimental results show that the proposed method achieved remarkable performance over the baseline and many advanced methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, S., Zhou, J., Sun, Y., et al.: An information minimization based contrastive learning model for unsupervised sentence embeddings learning. In: COLING, pp. 4821–4831 (2022)

    Google Scholar 

  2. Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. In: ICCV, pp. 9640–9649 (2021)

    Google Scholar 

  3. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR, pp. 539–546 (2005)

    Google Scholar 

  4. Choubey, P.K., Fabbri, A., Vig, J., et al.: CaPE: contrastive parameter ensembling for reducing hallucination in abstractive summarization. In: ACL, pp. 10755–10773 (2023)

    Google Scholar 

  5. Feng, J., Long, J., Han, C., et al.: RepSum: a general abstractive summarization framework with dynamic word embedding representation correction. Comput. Speech Lang. 80, 101491 (2023)

    Article  Google Scholar 

  6. Hermann, K.M., Kočiský, T., Grefenstette, E., et al.: Teaching machines to read and comprehend. In: NIPS, pp. 1693–1701 (2015)

    Google Scholar 

  7. Hou, C., Li, Z., Tang, Z., et al.: Multiple instance relation graph reasoning for cross-modal hash retrieval. Knowl.-Based Syst. 256, 109891 (2022)

    Article  Google Scholar 

  8. Hsu, W.T., Lin, C.K., Lee, M.Y., et al.: A unified model for extractive and abstractive summarization using inconsistency loss. In: ACL, pp. 132–141 (2018)

    Google Scholar 

  9. Lebanoff, L., Song, K., Dernoncourt, F., et al.: Scoring sentence singletons and pairs for abstractive summarization. In: ACL, pp. 2175–2189 (2019)

    Google Scholar 

  10. Lewis, M., Liu, Y., Goyal, N., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL, pp. 7871–7880 (2020)

    Google Scholar 

  11. Li, Z., Peng, Z., Tang, S., et al.: Text summarization method based on double attention pointer network. IEEE Access 8, 11279–11288 (2020)

    Article  Google Scholar 

  12. Li, Z., Sun, Y., Zhu, J., et al.: Improve relation extraction with dual attention-guided graph convolutional networks. Neural Comput. Appl. 33, 1773–1784 (2021)

    Article  Google Scholar 

  13. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)

    Google Scholar 

  14. Liu, Y., Zhu, C., Zeng, M.: End-to-end segmentation-based news summarization. In: ACL, pp. 544–554 (2022)

    Google Scholar 

  15. Liu, Y., Liu, P.: SimCLS: a simple framework for contrastive learning of abstractive summarization. In: ACL, pp. 1065–1072 (2021)

    Google Scholar 

  16. Liu, Y., Liu, P., Radev, D., et al.: BRIO: bringing order to abstractive summarization. In: ACL, pp. 2890–2903 (2022)

    Google Scholar 

  17. Liu, Y., Jia, Q., Zhu, K.: Length control in abstractive summarization by pretraining information selection. In: ACL, pp. 6885–6895 (2022)

    Google Scholar 

  18. Nallapati, R., Zhou, B., dos Santos, C., et al.: Abstractive text summarization using sequence-to-sequence RNNs and Beyond. In: CoNLL, pp. 280–290 (2016)

    Google Scholar 

  19. Narayan, S., Cohen, S.B., Lapata, M.: Don’t give me the details, Just the Summary! Topic-Aware convolutional neural networks for extreme summarization. In: EMNLP, pp. 1797–1807 (2018)

    Google Scholar 

  20. Pang, R.Y., He, H.: Text generation by learning from demonstrations. In: ICLR, pp. 1–22 (2021)

    Google Scholar 

  21. Rush, A.M., Chopra, S., Weston, J.: A neural attention model for abstractive sentence summarization. In: EMNLP, pp. 379–389 (2015)

    Google Scholar 

  22. See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: ACL, pp. 1073–1083 (2017)

    Google Scholar 

  23. Shazeer, N., Stern, M.: Adafactor: adaptive learning rates with sublinear memory cost. In: ICML, pp. 4596–4604 (2018)

    Google Scholar 

  24. Sun, S., Li, W.: Alleviating exposure bias via contrastive learning for abstractive text summarization. arXiv preprint arXiv:2108.11846, pp. 1–6 (2021)

  25. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)

    Google Scholar 

  26. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: NIPS, pp. 6000–6010 (2017)

    Google Scholar 

  27. Wang, D., Chen, J., Zhou, H., et al.: Contrastive aligned joint learning for multilingual summarization. In: ACL, pp. 2739–2750 (2021)

    Google Scholar 

  28. Wang, F., Song, K., Zhang, H., et al.: Salience allocation as guidance for abstractive summarization. In: EMNLP, pp. 6094–6106 (2022)

    Google Scholar 

  29. Xian, T., Li, Z., Zhang, C., et al.: Dual global enhanced transformer for image captioning. Neural Netw. 148, 129–141 (2022)

    Article  Google Scholar 

  30. Xie, X., Li, Z., Tang, Z., et al.: Unifying knowledge iterative dissemination and relational reconstruction network for image-text matching. Inf. Process. Manage. 60(1), 103154 (2023)

    Article  Google Scholar 

  31. Xu, S., Zhang, X., Wu, Y., et al.: Sequence level contrastive learning for text summarization. In: AAAI, pp. 11556–11565 (2022)

    Google Scholar 

  32. Xu, S., Li, H., Yuan, P., et al.: Self-attention guided copy mechanism for abstractive summarization. In: ACL, pp. 1355–1362 (2020)

    Google Scholar 

  33. Zaheer, M., Guruganesh, G., Dubey, K.A., et al.: Big Bird: transformers for longer sequences. In: NIPS, pp. 17283–17297 (2020)

    Google Scholar 

  34. Zhang, J., Zhao, Y., Saleh, M., et al.: PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In: ICML, pp. 11328–11339 (2020)

    Google Scholar 

  35. Zheng, C., Zhang, K., Wang, H.J., et al.: Enhanced Seq2Seq autoencoder via contrastive learning for abstractive text summarization. In: IEEE Big Data, pp. 1764–1771 (2021)

    Google Scholar 

  36. Zhong, M., Liu, P., Chen, Y., et al.: Extractive summarization as text matching. In: ACL, pp. 6197–6208 (2020)

    Google Scholar 

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China (Nos. 62276073, 61966004), Guangxi Natural Science Foundation (No. 2019GXNSFDA245018), Innovation Project of Guangxi Graduate Education (YCSW2023141), Guangxi “Bagui Scholar” Teams for Innovation and Research Project, and Guangxi Collaborative Innovation Center of Multi-source Information Integration and Intelligent Processing.

Author information

Authors and Affiliations

  1. Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China

    Ying Huang & Zhixin Li

  2. Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China

    Ying Huang & Zhixin Li

Authors
  1. Ying Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhixin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhixin Li .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Fenrong Liu

  2. SEEK Limited, Cremorne, NSW, Australia

    Arun Anand Sadanandan

  3. MIMOS (Malaysia), Kuala Lumpur, Malaysia

    Duc Nghia Pham

  4. Universitas Indonesia, Depok, Indonesia

    Petrus Mursanto

  5. Tabcorp Holdings Limited, Melbourne, VIC, Australia

    Dickson Lukose

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, Y., Li, Z. (2024). A Semantic Similarity Distance-Aware Contrastive Learning for Abstractive Summarization. In: Liu, F., Sadanandan, A.A., Pham, D.N., Mursanto, P., Lukose, D. (eds) PRICAI 2023: Trends in Artificial Intelligence. PRICAI 2023. Lecture Notes in Computer Science(), vol 14325. Springer, Singapore. https://doi.org/10.1007/978-981-99-7019-3_18

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-7019-3_18

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-7018-6

  • Online ISBN: 978-981-99-7019-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics