Abstract
A good video summary should convey the whole story and feature the most important content. However, the importance of video content is often subjective, and users should have the option to personalize the summary by using natural language to specify what is important to them. Moreover, existing methods usually apply only visual cues to solve generic video summarization tasks, while this work introduces a single unsupervised multi-modal framework for addressing both generic and query-focused video summarization. We use a multi-head attention model to represent the multi-modal feature. We apply a Transformer-based model to learn the frame scores based on their representative, diversity and reconstruction losses. Especially, we develop a novel representative loss to train the model based on both visual and semantic information. We outperform previous state-of-the-art work with superior results on both generic and query-focused video summarization datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cai, S., Zuo, W., Davis, L., Zhang, L.: Weakly-supervised video summarization using variational encoder-decoder and web prior. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, pp. 193–210. Springer International Publishing, Cham (2018)
Avila, S., Lopes, A., Antonio da Luz Jr., Araújo, A.: VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recognit. Lett. 32(1), 56–68 (2011)
Gong, B., Chao, W., Grauman, K., Sha F.: Diverse sequential subset selection for supervised video summarization. In: NIPS 2014, pp. 2069–2077. MIT Press, Cambridge, MA, USA, (2014)
Gygli, M., Grabner, H., Riemenschneider, H., Van Gool, L.: Creating Summaries from User Videos. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 505–520. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_33
He, X., et al.: Unsupervised video summarization with attentive conditional generative adversarial networks. In: ACM Multimedia 2019 (2019)
Iashin, V., Rahtu, E.: A better use of audio-visual cues: dense video captioning with bi-modal transformer. In: BMVC 2020 (2020)
Lee, Y., Ghosh, J., Grauman, K.: Discovering important people and objects for egocentric video summarization. In: CVPR 2012, pp. 1346–1353 (2012)
Lei, Z., Sun, K., Zhang, Q., Qiu, G.: User video summarization based on joint visual and semantic affinity graph. In: ACM Multimedia 2016 Workshop on Vision and Language Integration Meets Multimedia Fusion, pp. 45–52 (2016)
Li, Y., Wang, L., Yang, T., Gong, B.: How Local Is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 156–174. Springer, Cham (2018)
Mahasseni, B., Lam, M., Todorovic, S.: Unsupervised video summarization with adversarial LSTM networks. In: CVPR 2017, pp. 2982–2991 (2017)
Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: Conference on Empirical Methods in Natural Language Processing 2004, pp. 404–411 (2004)
Narasimhan, M., Rohrbach, A., Darrell, T.: CLIP-It! Language-guided video summarization. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J. (eds.) NIPS 2021, pp. 13988–14000 (2021)
Park, J., Lee, J., Kim, I.-J., Sohn, K.: SumGraph: Video Summarization via Recursive Graph Modeling. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 647–663. Springer, Cham (2020)
Rochan, M., Wang, Y.: Video summarization by learning from unpaired data. In: CVPR 2019, pp. 7902–7911 (2019)
Rochan, M., Ye, L., Wang, Y.: Video Summarization Using Fully Convolutional Sequence Networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11216, pp. 358–374. Springer, Cham (2018)
Sharghi, A., Gong, B., Shah, M.: Query-Focused Extractive Video Summarization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 3–19. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_1
Sharghi, A., Laurel, J., Gong, B.: Query-focused video summarization: dataset, evaluation, and a memory network based approach. In: CVPR 2017, pp. 2127–2136 (2017)
Song, Y., Vallmitjana, J., Stent, A., Jaimes, A.: TVSum: summarizing web videos using titles. In: CVPR 2015, pp. 5179–5187 (2015)
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) NIPS 2017, pp. 5998–6008, Long Beach, CA, USA (2017)
Yuan, L., Tay, F., Li, P., Zhou, L., Feng, J.: Cycle-SUM: cycle-consistent adversarial LSTM networks for unsupervised video summarization. In: AAAI 2019 (2019)
Zhang, K., Chao, W., Sha, F., Grauman, K.: Summary transfer: exemplar-based subset selection for video summarization. In: CVPR 2016 (2016)
Zhang, K., Chao, W.-L., Sha, F., Grauman, K.: Video Summarization with Long Short-Term Memory. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 766–782. Springer, Cham (2016)
Zhang, K., Grauman, K., Sha, F.: Retrospective Encoders for Video Summarization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 391–408. Springer, Cham (2018)
Zhang, Q., Qiu, G.: Bundling centre for landmark image discovery. In: ICMR 2015, pp. 179–186 (2015)
Zhao, B., Li, X., Lu, X.: Hierarchical recurrent neural network for video summarization. In: Liu, Q., et al. (eds.) ACMM 2017, pp. 863–871 (2017)
Zhao, B., Li, X., Xiaoqiang, L.: TTH-RNN: tensor-train hierarchical recurrent neural network for video summarization. IEEE Trans. Ind. Electron. 68(4), 3629–3637 (2021)
Zhou, K., Qiao Y.: Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward. In: AAAI 2017 (2017)
Saquil, Y., Chen, D., He, Y., Li, C., Yang, Y.-L.: Multiple pairwise ranking networks for personalized video summarization. In: ICCV 2021, pp. 1718–1727 (2021)
Lei, J., Berg, T.L., Bansal, M.: QVHightlights: detecting moments and highlights in videos via natural language queries. In: NIPS 2021 (2021)
Liu, Y., Li, S., Wu, Y., Chen, C., Shan, Y., Qie, X.: Umt: unified multimodal transformers for joint video moment retrieval and highlight detection. In: CVPR 2022 (2022)
Wu, G., Lin, J., Silva, C.T.: Intentvizor: towards generic query guided interactive video summarization. In: CVPR 2022, pp. 10503–10512 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lei, Z., Yu, Q., Shou, L., Li, S., Mao, Y. (2023). An Unsupervised Video Summarization Method Based on Multimodal Representation. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science(), vol 14090. Springer, Singapore. https://doi.org/10.1007/978-981-99-4761-4_15
Download citation
DOI: https://doi.org/10.1007/978-981-99-4761-4_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4760-7
Online ISBN: 978-981-99-4761-4
eBook Packages: Computer ScienceComputer Science (R0)