Skip to main content
Springer Nature Link
Log in

Convolutional neural network based low complexity HEVC intra encoder

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Video coding is one of the key technologies of visual sensors. As the state-of-art video coding standard, High Efficiency Video Coding (HEVC) achieves a significant high compression ratio for video. However, it also introduces heavy computational complexity, leading to challenges in application of visual sensors. To reduce the complexity of HEVC intra encoder, this paper proposed a one-stage decision method of CU/PU partition and prediction mode for intra coding. First, the potential factors that may related to the corresponding decisions in CU/PU are explored. Based on this, a one-stage decision network (OSDN) structure is specially designed to determine these decisions. Consequently, the complexity of HEVC intra coding can be drastically reduced by avoiding the brute-force search. Then, OSDN is embedded into the HEVC reference software HM 15.0. Thresholds are set to let the encoder switch between OSDN and the original implementation in HEVC to obtain the final decisions. The experimental results show that the proposed method can reduce 73.69% intra encoding time with 0.1673 dB BD-PSNR loss on average. In addition, the trade-off between RD performance degradation and complexity reduction can be controlled by thresholds.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Bossen F, Bross B, Suhring K, Flynn D (2012) HEVC complexity and implementation analysis. IEEE Trans Circuits Syst Video Technol 22 (12):1685–1696

    Article  Google Scholar 

  2. Charfi Y, Wakamiya N, Murata M (2009) Challenging issues in visual sensor networks. IEEE Wirel Commun 16(2):44–49

    Article  Google Scholar 

  3. Chen G, Liu Z, Kobayashi T, Ikenaga T (2016) Multi-feature based fast depth decision in HEVC inter prediction for VLSI implementation. In: Proc CISP-BMEI, pp 124–128

  4. Correa G, Assuncao P, Agostini L, Da Silva Cruz LA (2012) Performance and computational complexity assessment of high-efficiency video encoders. IEEE Trans Circuits Syst Video Technol 22(12):1899–1909

    Article  Google Scholar 

  5. Correa G, Assuncao P, Da Silva Cruz LA, Agostini L (2014) Classification-based early termination for coding tree structure decision in HEVC. In: Proc ICECS, pp 239–242

  6. Feng Z, Liu P, Jia K, Duan K (2018) Fast intra CTU depth decision for HEVC. IEEE Access 6:45262–45269

    Article  Google Scholar 

  7. Gao Y, Wang D, Ji X, Liu Z, Yu X, Chen S (2016) CU partition mode decision for HEVC hardwired intra encoder using convolution neural network. IEEE Trans Image Process 25(11):5088–5103

    Article  MathSciNet  Google Scholar 

  8. Gao L, Dong S, Wang W, Wang R, Gao W (2015) Fast intra mode decision algorithm based on refinement in HEVC. In: Proc ISCAS, pp 517–520

  9. Garrido Abenza PP, Malumbres MP, Piñol P, López-Granado O (2018) Source coding options to improve HEVC video streaming in vehicular networks. Sensors 18(9):3107

    Article  Google Scholar 

  10. Gwon D, Choi H, Youn JM (2015) HEVC fast intra mode decision based on edge and SATD cost. In: Proc APMediaCast, pp 74–78

  11. Hu Y, Yang W, Xia S, Cheng W, Liu J (2018) Enhanced intra prediction with recurrent neural network in video coding. In: Proc DCC, pp 413–413

  12. Jiang X, Feng J, Song T, Katayama T (2019) Low-complexity and hardware-friendly H.265/HEVC encoder for vehicular ad-hoc networks. Sensors 19(8):1–15

    Article  Google Scholar 

  13. Katayama T, Kuroda K, Shi W, Song T, Shimamoto T (2018) Low-complexity intra coding algorithm based on convolutional neural network for HEVC. In: Proc ICICT, pp 115–118

  14. Kim YH, Kim TS, Sunwoo MH, Jeong JH (2016) Fast CU size decision method for HEVC using CU split information of adjacent frames. In: Proc ISOCC, pp 331–332

  15. Kim K, Ro WW (2018) Fast CU depth decision for HEVC using neural networks. IEEE Trans Circuits Syst Video Technol 14(8):1462–1473

    Google Scholar 

  16. Lainema J, Bossen F, Han WJ, Min J, Ugur K (2012) Intra coding of the HEVC standard. IEEE Trans Circuits Syst Video Technol 22(12):1792–1801

    Article  Google Scholar 

  17. Laude T, Ostermann J (2016) Deep learning-based intra prediction mode decision for HEVC. In: Proc PCS, pp 1–5

  18. Li Y, Liu Z, Ji X, Wang D (2018) CNN based CU partition mode decision algorithm for HEVC inter coding. In: Proc ICIP, pp 993–997

  19. Li F, Shuang F, Liu Z, Qian X (2018) A cost-constrained video quality satisfaction study on mobile devices. IEEE Trans Multimed 20 (5):1154–1168

    Article  Google Scholar 

  20. Li J, Li B, Xu J, Xiong R, Gao W (2018) Fully connected network-based intra prediction for image coding. IEEE Trans Image Process 27(7):3236–3247

    Article  MathSciNet  Google Scholar 

  21. Li Y, Li L, Li Z, Yang J, Xu N, Liu D, Li H (2018) A hybrid neural network for chroma intra prediction. In: Proc ICIP, pp 1797–1801

  22. Ruiz-coll D, Adzic V, Fernández-escribano G, Kalva H, Martínez J L, Cuenca P (2014) Fast partition algorithm for HEVC intra frame coding using machine learning. In: Proc ICIP, pp 4112–4116

  23. Shang X, Wang G, Fan T, Li Y (2015) Fast CU size decision and PU mode decision algorithm in HEVC intra coding. In: Proc ICIP, pp 1593–1597

  24. Shan Y, Yang EH (2017) Fast HEVC intra coding algorithm based on machine learning and Laplacian transparent composite model. In: Proc ICASSP, pp 2642–2646

  25. Song Y, Zeng Y, Li X, Cai B, Yang G (2017) Fast CU size decision and mode decision algorithm for intra prediction in HEVC. Multimed Tools Appl 76(2):2001–2017

    Article  Google Scholar 

  26. Sullivan GJ, Ohm JR, Han WJ, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22(12):1649–1668

    Article  Google Scholar 

  27. Wang T, Li F, Cosman PC (2018) H.265/HEVC video coding over lossy networks: flexible or fixed mode in one CTU? IEEE Access 6:71279–71284

    Article  Google Scholar 

  28. Wang X, Xue Y (2017) Fast HEVC inter prediction algorithm based on spatio-temporal block information. In: Proc IEEE BMSB, pp 1–5

  29. Xiph.org (2017) Xiph.org video test media. [Online] Available: https://media.xiph.org/video/derf/

  30. Xu M, Li T, Wang Z, Deng X, Yang R, Guan Z (2018) Reducing complexity of HEVC: a deep learning approach. IEEE Trans Image Process 27(20):5044–5059

    Article  MathSciNet  Google Scholar 

  31. Xu M, Li T, Wang Z, Deng X, Yang R, Guan Z (2017) HEVC-Complexity-reduction. [Online] available: https://github.com/tianyili2017/HEVC-complexity-reduction

  32. Yan S, Hong L, He W, Wang Q (2012) Gradient-based fast mode decision algorithm for intra prediction in HEVC. In: Proc SITIS, pp 225–229

  33. Yan N, Liu D, Li H, Li B, Li L, Wu F (2019) Convolutional neural network-based fractional-pixel motion compensation. IEEE Trans Circuits Syst Video Technol 29(3):840–853

    Article  Google Scholar 

  34. Yeh C, Zhang Z, Chen M, Lin C (2018) HEVC intra frame coding based on convolutional neural network. IEEE Access 6:50087–50095

    Article  Google Scholar 

  35. Zhang H, Ma Z (2014) Fast intra mode decision for high efficiency video coding (HEVC). IEEE Trans Circuits Syst Video Technol 24(4):660–668

    Article  Google Scholar 

  36. Zhang H, Song L, Luo Z, Yang X (2017) Learning a convolutional neural network for fractional interpolation in HEVC inter coding. In: Proc VCIP, pp 1–4

  37. Zhao Z, Wang S, Wang S, Zhang X, Ma S, Yang J (2018) Enhanced bi-prediction with convolutional neural network for high efficiency video coding. IEEE Trans Circuits Syst Video Technol 29(11):3291–3301

    Article  Google Scholar 

  38. Zhu L, Zhang Y, Pan Z, Wang R, Kwong S, Peng Z (2017) Binary and multi-class learning based low complexity optimization for HEVC encoding. IEEE Trans Broadcast 63(3):547–561

    Article  Google Scholar 

Download references

Acknowledgments

This research work was supported in part by the National Science Foundation of China (61671365), and the Key Research and Development Program of Shaanxi Province (2020KW-009).

Author information

Authors and Affiliations

  1. School of Information and Communications Engineering, Xi’an Jiaotong University, Xi’an, 710049, China

    Zixi Wang & Fan Li

  2. Guangdong Xi’an Jiaotong University Academy, Foshan, 528000, China

    Zixi Wang & Fan Li

Authors
  1. Zixi Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Fan Li
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Fan Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Li, F. Convolutional neural network based low complexity HEVC intra encoder. Multimed Tools Appl 80, 2441–2460 (2021). https://doi.org/10.1007/s11042-020-09231-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09231-8

Keywords