Low-latency video coding techniques

SONG Li; LIU Xiaoyong; WU Guoqing; ZHU Chen; HUANG Yan; XIE Rong; ZHANG Wenjun

doi:10.13700/j.bh.1001-5965.2020.0463

Volume 47 Issue 3

Mar. 2021

Turn off MathJax

Article Contents

Journal of Beijing University of Aeronautics and Astronautics > 2021 > 47(3): 558-571.

SONG Li, LIU Xiaoyong, WU Guoqing, et al. Low-latency video coding techniques[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 558-571. doi: 10.13700/j.bh.1001-5965.2020.0463(in Chinese)

Citation:

SONG Li, LIU Xiaoyong, WU Guoqing, et al. Low-latency video coding techniques[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 558-571. doi: 10.13700/j.bh.1001-5965.2020.0463(in Chinese)

Citation:

PDF( 3849 KB)

Low-latency video coding techniques

doi: 10.13700/j.bh.1001-5965.2020.0463

SONG Li^{1, 2
,
,},
LIU Xiaoyong¹,
WU Guoqing¹,
ZHU Chen¹,
HUANG Yan¹,
XIE Rong¹,
ZHANG Wenjun^{1, 2}

1.
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
2.
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China

Funds:

National Key R & D Program of China 2019YFB1802701

National Natural Science Foundation of China 61671296

More Information

Corresponding author: SONG Li, E-mail: song_li@sjtu.edu.cn
Received Date: 26 Aug 2020
Accepted Date: 19 Sep 2020
Publish Date: 20 Mar 2021

Abstract

Abstract

With the widespread usage of video coding and transmission techniques, demands for video have increased dramatically. Real-time video communication has become the research focus of the video industry. Its core goal is to provide a better user experience and lower latency. Low-latency video coding is a key component for real-time video communication applications. The overall system latency can be effectively reduced by reducing the coding latency. First, this paper analyzes sources of latency in the video transmission system. Focusing on the general video coding framework, this paper introduces the generation mechanism of the coding latency. Then, mainstream video coding standards at home and abroad are outlined. The detailed description of the principle and models of rate-distortion optimization techniques provides a theoretical basis for the design of low-latency video encoders. Additionally, this paper summarizes how to optimize the coding latency in terms of the reference structures, pipeline design, encoding modes search, rate control, and hardware acceleration and generalizes industrial representative low-latency video coding schemes. Finally, this paper summarizes the limitations of the existing low-latency video coding techniques and presents future research directions.
- low latency,
- video coding,
- real-time video communication,
- rate-distortion optimization,
- encoding use cases

FullText(HTML)

References(65)

References

[1]	CULLEN C. The global internet phenomena report[EB/OL]. [2020-08-26]. https://www.sandvine.com/press-releases/sandvine-releases-2019-global-internet-phenomena-report.
[2]	高文, 赵德斌, 马思伟. 数字视频编码技术原理[M]. 2版. 北京: 科学出版社, 2018: 17. GAO W, ZHAO D B, MA S W. Principles of digital video coding technology[M]. 2nd ed. Beijing: Science Press, 2018: 17(in Chinese).
[3]	SCHREIER R M, ROTHERMEL A. A latency analysis on H. 264 video transmission systems[C]//Proceedings of the 2008 Digest of Technical Papers-International Conference on Consumer Electronics. Piscataway: IEEE Press, 2008: 307-308.
[4]	SULLIVAN G J, WIEGAND T. Rate-distortion optimization for video compression[J]. IEEE Signal Processing Magazine, 1998, 15(6): 74-90. doi: 10.1109/79.733497
[5]	BICHON M, LE T J, ROPERTOPERT M, et al. Inter-block dependencies consideration for intra coding in H. 264/AVC and HEVC standards[C]//Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2017: 1537-1541.
[6]	BICHON M, LE T J, ROPERT M, et al. Low complexity joint RDO of prediction units couples for HEVC intra coding[C]//Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2018: 1733-1737.
[7]	PANG C, AU O C, ZOU F, et al. Optimal distortion redistribution in block-based image coding using successive convex optimization[C]//Proceedings of the 2011 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2011: 1-5.
[8]	WU Q, XIONG J, LUO B, et al. A novel joint rate distortion optimization scheme for intra prediction coding in H. 264/AVC[J]. IEICE Transactions on Information and Systems, 2014, 97(4): 989-992. http://ci.nii.ac.jp/naid/130003394928
[9]	YANG T, ZHU C, FAN X, et al. Source distortion temporal propagation model for motion compensated video coding optimization[C]//Proceedings of the 2012 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2012: 85-90.
[10]	LI S, ZHU C, GAO Y, et al. Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(1): 117-129. doi: 10.1109/TCSVT.2015.2450131
[11]	GONZÁLEZ-DE-SUSO J L, MARTÍNEZ-ENRÍQUEZ E, DÍAZ-DE-MARÍA F. Adaptive Lagrange multiplier estimation algorithm in HEVC[J]. Signal Processing: Image Communication, 2017, 56: 40-51. doi: 10.1016/j.image.2017.04.010
[12]	YANG K, WAN S, GONG Y, et al. An efficient Lagrangian multiplier selection method based on temporal dependency for rate-distortion optimization in H. 265/HEVC[J]. Signal Processing: Image Communication, 2017, 57: 68-75. doi: 10.1016/j.image.2017.05.006
[13]	ZHANG F, BULL D R. An adaptive Lagrange multiplier determination method for rate-distortion optimisation in hybrid video codecs[C]//Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2015: 671-675.
[14]	WANG X, SONG L, LUO Z, et al. Lagrangian method based rate-distortion optimization revisited for dependent video coding[C]//Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2017: 3021-3025.
[15]	LI C, WU D, XIONG H. Delay-power-rate-distortion model for wireless video communication under delay and energy constraints[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(7): 1170-1183. doi: 10.1109/TCSVT.2014.2302517
[16]	LI C, XIONG H, WU D. Delay-rate-distortion optimized rate control for wireless video communication[C]//Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2014: 5996-6000.
[17]	HUANG B, CHEN Z, CAI Q, et al. Rate-distortion-complexity optimized coding mode decision for HEVC[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(3): 795-809. doi: 10.1109/TCSVT.2019.2893396
[18]	CARBALLEIRA P, CABRERA J, ORTEGA A, et al. A graph-based approach for latency modeling and optimization in multiview video encoding[C]//Proceedings of the 20113DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON). Piscataway: IEEE Press, 2011: 1-4.
[19]	WENGER S. Temporal scalability using P-pictures for low-latency applications[C]//Proceedings of the 1998 IEEE Second Workshop on Multimedia Signal Processing. Piscataway: IEEE Press, 1998: 559-564.
[20]	PAN Z, JIN P, LEI J, et al. Fast reference frame selection based on content similarity for low complexity HEVC encoder[J]. Journal of Visual Communication and Image Representation, 2016, 40: 516-524. doi: 10.1016/j.jvcir.2016.07.018
[21]	PARK S H, DONG T, JANG E S. Low complexity reference frame selection in QTBT structure for JVET future video coding[C]//Proceedings of the 2018 International Workshop on Advanced Image Technology (IWAIT). Piscataway: IEEE Press, 2018: 1-4.
[22]	DESHPANDE S, HANNUKSELA M M, KAZUI K, et al. An improved hypothetical reference decoder for HEVC[C]//International Society for Optics and Photonics. Bellingham: SPIE-INT SOC Optical Engineering, 2013: 866608.
[23]	SCHREIER R M, RAHMAN A M T I, KRISHNA-MURTHY G, et al. Architecture analysis for low-delay video coding[C]//Proceedings of the 2006 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2006: 2053-2056.
[24]	MEENDERINCK C, AZEVEDO A, JUURLINK B, et al. Parallel scalability of video decoders[J]. Journal of Signal Processing Systems, 2009, 57(2): 173-194. doi: 10.1007/s11265-008-0256-9
[25]	CHI C C, ALVAREZ-MESA M, JUURLINK B, et al. Parallel scalability and efficiency of HEVC parallelization approaches[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1827-1838. doi: 10.1109/TCSVT.2012.2223056
[26]	MOHAMED M, NEJMEDDINE B, NOUREDDINE B, et al. Performance evaluation of frame-level parallelization in HEVC intra coding using heterogeneous multicore platforms[C]//Proceedings of the 2018 International Conference on Applied Smart Systems (ICASS). Piscataway: IEEE Press, 2018: 1-6.
[27]	CHEN K, SUN J, DUAN Y, et al. A novel wavefront-based high parallel solution for HEVC encoding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(1): 181-194. doi: 10.1109/TCSVT.2015.2418651
[28]	WANG H, XIAO B, WU J, et al. A collaborative scheduling-based parallel solution for HEVC encoding on multicore platforms[J]. IEEE Transactions on Multimedia, 2018, 20(11): 2935-2948. doi: 10.1109/TMM.2018.2830120
[29]	OHM J R, SULLIVAN G J, SCHWARZ H, et al. Comparison of the coding efficiency of video coding standards-Including high efficiency video coding (HEVC)[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1669-1684. doi: 10.1109/TCSVT.2012.2221192
[30]	BOSSEN F, BROSS B, SUHRING K, et al. HEVC complexity and implementation analysis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1685-1696. doi: 10.1109/TCSVT.2012.2221255
[31]	TOPIWALA P, KRISHNAN M, DAI W. Performance comparison of VVC, AV1 and EVC[C]//Proceedings of Applications of Digital Image Processing XLⅡ. Bellingham: SPIE-INT SOC Optical Engineering, 2019.
[32]	SULLIVAN G J, OHM J R, HAN W J, et al. Overview of the high efficiency video coding (HEVC) standard[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1649-1668. doi: 10.1109/TCSVT.2012.2221191
[33]	ZHANG T, SUN M T, ZHAO D, et al. Fast intra-mode and CU size decision for HEVC[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(8): 1714-1726. doi: 10.1109/TCSVT.2016.2556518
[34]	MIN B, CHEUNG R C C. A fast CU size decision algorithm for the HEVC intra encoder[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(5): 892-896. doi: 10.1109/TCSVT.2014.2363739
[35]	XIONG J, LI H, WU Q, et al. A fast HEVC inter CU selection method based on pyramid motion divergence[J]. IEEE Transactions on Multimedia, 2014, 16(2): 559-564. doi: 10.1109/TMM.2013.2291958
[36]	PAN Z, KWONG S, SUN M T, et al. Early MERGE mode decision based on motion estimation and hierarchical depth correlation for HEVC[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 405-412. doi: 10.1109/TBC.2014.2321682
[37]	LIU Z, YU X, GAO Y, et al. CU partition mode decision for HEVC hardwired intra encoder using convolution neural network[J]. IEEE Transactions on Image Processing, 2016, 25(11): 5088-5103. doi: 10.1109/TIP.2016.2601264
[38]	RYU S, KANG J. Machine learning-based fast angular prediction mode decision technique in video coding[J]. IEEE Transactions on Image Processing, 2018, 27(11): 5525-5538. doi: 10.1109/TIP.2018.2857404
[39]	XU M, LI T, WANG Z, et al. Reducing complexity of HEVC: A deep learning approach[J]. IEEE Transactions on Image Processing, 2017, 27(10): 5044-5059. http://ieeexplore.ieee.org/abstract/document/8384310/
[40]	AMESTOY T, MERCAT A, HAMIDOUCHE W, et al. Tunable VVC frame partitioning based on lightweight machine learning[J]. IEEE Transactions on Image Processing, 2019, 29: 1313-1328. http://ieeexplore.ieee.org/document/8826595
[41]	TRAN T D, LIU L K, WESTERINK P H. Low-delay MPEG-2 video coding[C]//Proceedings of the SPIE-The International Society for Optical Engineering. Bellingham: SPIE-INT SOC Optical Engineering, 1997: 510-516.
[42]	TOURAPIS A M, LEONTARIS A, SUHRING K, et al. H. 264/14496-10 AVC reference software manual[EB/OL]. [2020-07-04]. http://iphone.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%(JVT-AE010).pdf.
[43]	DELA CRUZ A R, CAJOTE R D. Low complexity adaptive intra-refresh rate for real-time wireless video transmission[C]//Proceedings of the Signal and Information Processing Association Annual Summit and Conference (APSIPA). Piscataway: IEEE Press, 2014: 1-5.
[44]	ZHOU Y R, LI G Q, NING S S. A new feedback-based intra refresh method for robust video coding[C]//Proceedings of the 2015 International Conference on Computer Science and Applications (CSA). Piscataway: IEEE Press, 2015: 218-221.
[45]	HENDRY, WANG Y K, CHEN J L, et al. CE11/AHG14: Test 3.1-An approach for support of GRA/GDR[EB/OL]. [2020-07-04]. http://phenix.it-sudparis.eu/jvet/doc_end_user/documents/15_Gothenburg/wg11/JVET-O124-v3.zip.
[46]	LEE Y G, SONG B C. An intra-frame rate control algorithm for ultralow delay H. 264/advanced video coding (AVC)[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(5): 747-752. doi: 10.1109/TCSVT.2009.2017413
[47]	CHOI H, NAM J, YOO J, et al. Rate control based on unified RQ model for HEVC[EB/OL]. [2020-07-04]. http://phenix.int-evry.fr/jct/doc_end_user/documents/8_San%20Jose/wg11/JCTVC-H0213-v3.zip.
[48]	CHOI H, YOO J, NAM J, et al. Pixel-wise unified rate-quantization model for multi-level rate control[J]. IEEE Journal of Selected Topics in Signal Processing, 2013, 7(6): 1112-1123. doi: 10.1109/JSTSP.2013.2272241
[49]	WANG S, MA S, WANG S, et al. Quadratic ρ-domain based rate control algorithm for HEVC[C]//Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processin(ICASSP). Piscataway: IEEE Press, 2013: 1695-1699.
[50]	LI B, LI H, LI L, et al. λ domain rate control algorithm for high efficiency video coding[J]. IEEE Transactions on Image Processing, 2014, 23(9): 3841-3854. doi: 10.1109/TIP.2014.2336550
[51]	GAO W, KWONG S, JIA Y. Joint machine learning and game theory for rate control in high efficiency video coding[J]. IEEE Transactions on Image Processing, 2017, 26(12): 6074-6089. doi: 10.1109/TIP.2017.2745099
[52]	KWONG S, ZHOU M, XUEKAI W E I, et al. Rate control method based on deep reinforcement learning for dynamic video sequences in HEVC[J]. IEEE Transactions on Multimedia, 2020, 99: 1. http://www.researchgate.net/publication/341198848_Rate_Control_Method_Based_on_Deep_Reinforcement_Learning_for_Dynamic_Video_Sequences_in_HEVC
[53]	JIANG C, NOOSHABADI S. A scalable massively parallel motion and disparity estimation scheme for multiview video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(2): 346-359. doi: 10.1109/TCSVT.2015.2402853
[54]	SHAHID M U, AHMED A, MARTINA M, et al. Parallel H. 264/AVC fast rate-distortion optimized motion estimation by using a graphics processing unit and dedicated hardware[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(4): 701-715. doi: 10.1109/TCSVT.2014.2351111
[55]	XIAO B, WANG H, WU J, et al. A multi-grained parallel solution for HEVC encoding on heterogeneous platforms[J]. IEEE Transactions on Multimedia, 2019, 21(12): 2997-3009. doi: 10.1109/TMM.2019.2916462
[56]	MOMCILOVIC S, ROMA N, SOUSA L. Exploiting task and data parallelism for advanced video coding on hybrid CPU + GPU platforms[J]. Journal of Real-Time Image Processing, 2016, 11(3): 571-587. doi: 10.1007/s11554-013-0357-y
[57]	INTEL. Scalable video technology for HEVC encoder (SVT-HEVC Encoder)[EB/OL]. (2020-08-20)[2020-08-26]. https://github.com/OpenVisualCloud/SVT-HEVC.
[58]	SURUR. TPCAST's wireless Adapter coming to the Oculus Rift next week[EB/OL]. (2017-11-08)[2020-08-26]. https://mspoweruser.com/tpcasts-wireless-adapter-coming-oculus-rift-next-week/.
[59]	BUYSSCHAERT C, DESCAMPE A, FÖßEL S, et al. Overview of JPEG XS[EB/OL]. 2018[2020-08-26]. https://jpeg.org/jpegxs/index.html.
[60]	AMIMON, Ltd. Wireless home digital interface^TM specification v1.0 Revision 33[EB/OL]. [2020-07-04]. https://www.amimon.com/proavl.
[61]	NETINT. Codensity T408 video transcoder product brief[EB/OL]. [2020-08-26]. https://www.netint.ca/product/t400_transcoder/.
[62]	INTEL. Scalable video technology for the visual cloud[EB/OL]. [2020-08-26]. https://01.org/sites/default/files/documentation/svt_aws_wp.pdf.
[63]	CAST. H264-E-BPF ultra-fast AVC/H. 264 baseline profile encoder[EB/OL]. [2020-08-26]. https://www.cast-inc.com/compression/avc-hevc-video-compression/h264-e-bpf/.
[64]	RICHTER T, KEINERT J, FOESSEL S, et al. JPEG-XS-A high-quality mezzanine image codec for video over IP[J]. SMPTE Motion Imaging Journal, 2018, 127(9): 39-49. doi: 10.5594/JMI.2018.2862098
[65]	REZNIC Z. Successive refinement video compression: USA, WO2018025211A1[P]. 2018-02-08.

Relative Articles

Supplements(0)

Cited By

Proportional views

Proportional views

通讯作者: 陈斌, bchen63@163.com

1.
沈阳化工大学材料科学与工程学院沈阳 110142

Figures(7) / Tables(2)

Get Citation

PDF

XML

Article Metrics

Article views(776) PDF downloads(109)

Low-latency video coding techniques

doi: 10.13700/j.bh.1001-5965.2020.0463

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Low-latency video coding techniques

doi: 10.13700/j.bh.1001-5965.2020.0463

Abstract

References

Proportional views

Catalog

通讯作者: 陈斌, bchen63@163.com

Article Metrics

Proportional views

Related

Export File

Citation

Format

Content