低延迟视频编码技术

宋利; 刘孝勇; 武国庆; 朱辰; 黄琰; 解蓉; 张文军

doi:10.13700/j.bh.1001-5965.2020.0463

低延迟视频编码技术

doi: 10.13700/j.bh.1001-5965.2020.0463

宋利^{1, 2, ,},
刘孝勇¹,
武国庆¹,
朱辰¹,
黄琰¹,
解蓉¹,
张文军^{1, 2}

1.
上海交通大学图像通信与网络工程研究所, 上海 200240
2.
上海交通大学未来媒体网络协同创新中心, 上海 200240

基金项目:

国家重点研发计划 2019YFB1802701

国家自然科学基金 61671296

详细信息

作者简介:
宋利  男，博士，教授，博士生导师。主要研究方向：新型视频编码、大数据压缩、移动计算视觉

刘孝勇  男，博士研究生。主要研究方向：可伸缩视频编码、码率控制

武国庆  男，硕士研究生。主要研究方向：视频编码

朱辰  男，博士研究生。主要研究方向：视频编码

黄琰  男，博士研究生。主要研究方向：视频编码

解蓉  女，博士，副教授，硕士生导师。主要研究方向：视频编码与转码、图像/视频处理

张文军  男，博士，教授，博士生导师。主要研究方向：图像通信与数字电视、宽带无线传输、系统芯片设计

通讯作者:
宋利, E-mail: song_li@sjtu.edu.cn

中图分类号: TN919.8
计量
- 文章访问数: 752
- HTML全文浏览量: 92
- PDF下载量: 106
- 被引次数: 0
出版历程
- 收稿日期: 2020-08-26
- 录用日期: 2020-09-19
- 网络出版日期: 2021-03-20

Low-latency video coding techniques

SONG Li^{1, 2
, ,},
LIU Xiaoyong¹,
WU Guoqing¹,
ZHU Chen¹,
HUANG Yan¹,
XIE Rong¹,
ZHANG Wenjun^{1, 2}

1.
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
2.
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China

Funds:

National Key R & D Program of China 2019YFB1802701

National Natural Science Foundation of China 61671296

More Information

Corresponding author: SONG Li, E-mail: song_li@sjtu.edu.cn

摘要

摘要:
随着视频编码和视频传输技术的广泛应用，视频需求量剧增，实时视频通信成为视频行业的一项重要研究内容，核心目标是提供更好的用户体验和更低的延迟。低延迟视频编码是实时视频通信应用的关键部分，通过降低编码延迟可以有效地降低系统的整体延迟。首先，分析了视频传输系统的延迟来源，从通用的视频编码框架出发着重介绍了编码延迟的产生机制；其次，概述了国内外主流的视频编码标准，介绍了率失真优化技术的原理和模型，为低延迟视频编码器的设计提供了理论基础；最后，从参考结构、流水线设计、编码模式搜索、码率控制和硬件加速多个维度描述了优化编码延迟的技术手段，并总结了业界具有代表性的低延迟视频编码方案，简要说明了现有低延迟视频编码技术的局限性，并对未来的发展方向做了展望。
- 低延迟 /
- 视频编码 /
- 实时视频通信 /
- 率失真优化 /
- 编码用例
Abstract:
With the widespread usage of video coding and transmission techniques, demands for video have increased dramatically. Real-time video communication has become the research focus of the video industry. Its core goal is to provide a better user experience and lower latency. Low-latency video coding is a key component for real-time video communication applications. The overall system latency can be effectively reduced by reducing the coding latency. First, this paper analyzes sources of latency in the video transmission system. Focusing on the general video coding framework, this paper introduces the generation mechanism of the coding latency. Then, mainstream video coding standards at home and abroad are outlined. The detailed description of the principle and models of rate-distortion optimization techniques provides a theoretical basis for the design of low-latency video encoders. Additionally, this paper summarizes how to optimize the coding latency in terms of the reference structures, pipeline design, encoding modes search, rate control, and hardware acceleration and generalizes industrial representative low-latency video coding schemes. Finally, this paper summarizes the limitations of the existing low-latency video coding techniques and presents future research directions.
- low latency /
- video coding /
- real-time video communication /
- rate-distortion optimization /
- encoding use cases

HTML全文

图 1 HEVC的RA模式的参考关系

Figure 1. Reference structure of RA mode for HEVC

下载: 全尺寸图片幻灯片

图 2 HEVC的LDP模式的参考关系

Figure 2. Reference structure of LDP mode for HEVC

下载: 全尺寸图片幻灯片

图 3 帧级编码和条/宏块级编码的端到端延迟^[22]

Figure 3. End-to-end latency of frame-level encoding and slice/macroblock-level encoding^[22]

下载: 全尺寸图片幻灯片

图 4 帧、条、块级编码任务分解^[23]

Figure 4. Encoding processes at frame, slice and block levels^[23]

下载: 全尺寸图片幻灯片

图 5 片级并行和波前并行处理^[22]

Figure 5. Tile parallelization and wavefront parallel processing^[22]

下载: 全尺寸图片幻灯片

图 6 刷新和未刷新区域的编码块参考^[45]

Figure 6. Encoding block reference structure for refreshed and unrefreshed regions^[45]

下载: 全尺寸图片幻灯片

图 7 编码方案对比

Figure 7. Comparison of different encoding schemes

下载: 全尺寸图片幻灯片

表 1 不同方法的码率控制精度对比^[52]

Table 1. Rate control accuracy comparison of different methods^[52]

序列	HM16.19		文献[51]		文献[52]
序列	F/%	Seq/%	F/%	Seq/%	F/%	Seq/%
Mobisode	2.55	0.010	1.98	0.010	1.01	0.003
RaceHorses	1.53	0.008	1.43	0.007	0.66	0.004
PartyScene	2.33	0.009	2.11	0.008	0.29	0.003
Chromakey	2.55	0.010	2.32	0.009	0.33	0.003
Wave	2.22	0.009	2.01	0.008	0.28	0.002
Parkrun	2.73	0.011	2.38	0.009	0.41	0.003
Kimono	1.22	0.007	1.34	0.008	0.67	0.004
Tennis	2.76	0.011	2.41	0.009	0.44	0.004
Average	2.24	0.009	2.00	0.009	0.51	0.003

下载: 导出CSV

表 2 编码方案配置

Table 2. Configuration of different encoding schemes

	编码方案	SVT-HEVC	TPCast	JPEG-XS	WHDI
编码模块	帧间/帧内预测	√	√	×	×
	变换	√	√	√	√
	量化	√	√	√	√
	熵编码	√	√	极简	×
延迟优化技术	参考关系简化	√	√	×	×
	硬件加速	√	√	√	×
	帧内刷新	×	√	×	×
	并行	√	×	√	×

下载: 导出CSV

参考文献(65)

[1]	CULLEN C. The global internet phenomena report[EB/OL]. [2020-08-26]. https://www.sandvine.com/press-releases/sandvine-releases-2019-global-internet-phenomena-report.
[2]	高文, 赵德斌, 马思伟. 数字视频编码技术原理[M]. 2版. 北京: 科学出版社, 2018: 17. GAO W, ZHAO D B, MA S W. Principles of digital video coding technology[M]. 2nd ed. Beijing: Science Press, 2018: 17(in Chinese).
[3]	SCHREIER R M, ROTHERMEL A. A latency analysis on H. 264 video transmission systems[C]//Proceedings of the 2008 Digest of Technical Papers-International Conference on Consumer Electronics. Piscataway: IEEE Press, 2008: 307-308.
[4]	SULLIVAN G J, WIEGAND T. Rate-distortion optimization for video compression[J]. IEEE Signal Processing Magazine, 1998, 15(6): 74-90. doi: 10.1109/79.733497
[5]	BICHON M, LE T J, ROPERTOPERT M, et al. Inter-block dependencies consideration for intra coding in H. 264/AVC and HEVC standards[C]//Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2017: 1537-1541.
[6]	BICHON M, LE T J, ROPERT M, et al. Low complexity joint RDO of prediction units couples for HEVC intra coding[C]//Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2018: 1733-1737.
[7]	PANG C, AU O C, ZOU F, et al. Optimal distortion redistribution in block-based image coding using successive convex optimization[C]//Proceedings of the 2011 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2011: 1-5.
[8]	WU Q, XIONG J, LUO B, et al. A novel joint rate distortion optimization scheme for intra prediction coding in H. 264/AVC[J]. IEICE Transactions on Information and Systems, 2014, 97(4): 989-992. http://ci.nii.ac.jp/naid/130003394928
[9]	YANG T, ZHU C, FAN X, et al. Source distortion temporal propagation model for motion compensated video coding optimization[C]//Proceedings of the 2012 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2012: 85-90.
[10]	LI S, ZHU C, GAO Y, et al. Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(1): 117-129. doi: 10.1109/TCSVT.2015.2450131
[11]	GONZÁLEZ-DE-SUSO J L, MARTÍNEZ-ENRÍQUEZ E, DÍAZ-DE-MARÍA F. Adaptive Lagrange multiplier estimation algorithm in HEVC[J]. Signal Processing: Image Communication, 2017, 56: 40-51. doi: 10.1016/j.image.2017.04.010
[12]	YANG K, WAN S, GONG Y, et al. An efficient Lagrangian multiplier selection method based on temporal dependency for rate-distortion optimization in H. 265/HEVC[J]. Signal Processing: Image Communication, 2017, 57: 68-75. doi: 10.1016/j.image.2017.05.006
[13]	ZHANG F, BULL D R. An adaptive Lagrange multiplier determination method for rate-distortion optimisation in hybrid video codecs[C]//Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2015: 671-675.
[14]	WANG X, SONG L, LUO Z, et al. Lagrangian method based rate-distortion optimization revisited for dependent video coding[C]//Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2017: 3021-3025.
[15]	LI C, WU D, XIONG H. Delay-power-rate-distortion model for wireless video communication under delay and energy constraints[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(7): 1170-1183. doi: 10.1109/TCSVT.2014.2302517
[16]	LI C, XIONG H, WU D. Delay-rate-distortion optimized rate control for wireless video communication[C]//Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE Press, 2014: 5996-6000.
[17]	HUANG B, CHEN Z, CAI Q, et al. Rate-distortion-complexity optimized coding mode decision for HEVC[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(3): 795-809. doi: 10.1109/TCSVT.2019.2893396
[18]	CARBALLEIRA P, CABRERA J, ORTEGA A, et al. A graph-based approach for latency modeling and optimization in multiview video encoding[C]//Proceedings of the 20113DTV Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON). Piscataway: IEEE Press, 2011: 1-4.
[19]	WENGER S. Temporal scalability using P-pictures for low-latency applications[C]//Proceedings of the 1998 IEEE Second Workshop on Multimedia Signal Processing. Piscataway: IEEE Press, 1998: 559-564.
[20]	PAN Z, JIN P, LEI J, et al. Fast reference frame selection based on content similarity for low complexity HEVC encoder[J]. Journal of Visual Communication and Image Representation, 2016, 40: 516-524. doi: 10.1016/j.jvcir.2016.07.018
[21]	PARK S H, DONG T, JANG E S. Low complexity reference frame selection in QTBT structure for JVET future video coding[C]//Proceedings of the 2018 International Workshop on Advanced Image Technology (IWAIT). Piscataway: IEEE Press, 2018: 1-4.
[22]	DESHPANDE S, HANNUKSELA M M, KAZUI K, et al. An improved hypothetical reference decoder for HEVC[C]//International Society for Optics and Photonics. Bellingham: SPIE-INT SOC Optical Engineering, 2013: 866608.
[23]	SCHREIER R M, RAHMAN A M T I, KRISHNA-MURTHY G, et al. Architecture analysis for low-delay video coding[C]//Proceedings of the 2006 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE Press, 2006: 2053-2056.
[24]	MEENDERINCK C, AZEVEDO A, JUURLINK B, et al. Parallel scalability of video decoders[J]. Journal of Signal Processing Systems, 2009, 57(2): 173-194. doi: 10.1007/s11265-008-0256-9
[25]	CHI C C, ALVAREZ-MESA M, JUURLINK B, et al. Parallel scalability and efficiency of HEVC parallelization approaches[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1827-1838. doi: 10.1109/TCSVT.2012.2223056
[26]	MOHAMED M, NEJMEDDINE B, NOUREDDINE B, et al. Performance evaluation of frame-level parallelization in HEVC intra coding using heterogeneous multicore platforms[C]//Proceedings of the 2018 International Conference on Applied Smart Systems (ICASS). Piscataway: IEEE Press, 2018: 1-6.
[27]	CHEN K, SUN J, DUAN Y, et al. A novel wavefront-based high parallel solution for HEVC encoding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(1): 181-194. doi: 10.1109/TCSVT.2015.2418651
[28]	WANG H, XIAO B, WU J, et al. A collaborative scheduling-based parallel solution for HEVC encoding on multicore platforms[J]. IEEE Transactions on Multimedia, 2018, 20(11): 2935-2948. doi: 10.1109/TMM.2018.2830120
[29]	OHM J R, SULLIVAN G J, SCHWARZ H, et al. Comparison of the coding efficiency of video coding standards-Including high efficiency video coding (HEVC)[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1669-1684. doi: 10.1109/TCSVT.2012.2221192
[30]	BOSSEN F, BROSS B, SUHRING K, et al. HEVC complexity and implementation analysis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1685-1696. doi: 10.1109/TCSVT.2012.2221255
[31]	TOPIWALA P, KRISHNAN M, DAI W. Performance comparison of VVC, AV1 and EVC[C]//Proceedings of Applications of Digital Image Processing XLⅡ. Bellingham: SPIE-INT SOC Optical Engineering, 2019.
[32]	SULLIVAN G J, OHM J R, HAN W J, et al. Overview of the high efficiency video coding (HEVC) standard[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(12): 1649-1668. doi: 10.1109/TCSVT.2012.2221191
[33]	ZHANG T, SUN M T, ZHAO D, et al. Fast intra-mode and CU size decision for HEVC[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(8): 1714-1726. doi: 10.1109/TCSVT.2016.2556518
[34]	MIN B, CHEUNG R C C. A fast CU size decision algorithm for the HEVC intra encoder[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(5): 892-896. doi: 10.1109/TCSVT.2014.2363739
[35]	XIONG J, LI H, WU Q, et al. A fast HEVC inter CU selection method based on pyramid motion divergence[J]. IEEE Transactions on Multimedia, 2014, 16(2): 559-564. doi: 10.1109/TMM.2013.2291958
[36]	PAN Z, KWONG S, SUN M T, et al. Early MERGE mode decision based on motion estimation and hierarchical depth correlation for HEVC[J]. IEEE Transactions on Broadcasting, 2014, 60(2): 405-412. doi: 10.1109/TBC.2014.2321682
[37]	LIU Z, YU X, GAO Y, et al. CU partition mode decision for HEVC hardwired intra encoder using convolution neural network[J]. IEEE Transactions on Image Processing, 2016, 25(11): 5088-5103. doi: 10.1109/TIP.2016.2601264
[38]	RYU S, KANG J. Machine learning-based fast angular prediction mode decision technique in video coding[J]. IEEE Transactions on Image Processing, 2018, 27(11): 5525-5538. doi: 10.1109/TIP.2018.2857404
[39]	XU M, LI T, WANG Z, et al. Reducing complexity of HEVC: A deep learning approach[J]. IEEE Transactions on Image Processing, 2017, 27(10): 5044-5059. http://ieeexplore.ieee.org/abstract/document/8384310/
[40]	AMESTOY T, MERCAT A, HAMIDOUCHE W, et al. Tunable VVC frame partitioning based on lightweight machine learning[J]. IEEE Transactions on Image Processing, 2019, 29: 1313-1328. http://ieeexplore.ieee.org/document/8826595
[41]	TRAN T D, LIU L K, WESTERINK P H. Low-delay MPEG-2 video coding[C]//Proceedings of the SPIE-The International Society for Optical Engineering. Bellingham: SPIE-INT SOC Optical Engineering, 1997: 510-516.
[42]	TOURAPIS A M, LEONTARIS A, SUHRING K, et al. H. 264/14496-10 AVC reference software manual[EB/OL]. [2020-07-04]. http://iphone.hhi.de/suehring/tml/JM%20Reference%20Software%20Manual%(JVT-AE010).pdf.
[43]	DELA CRUZ A R, CAJOTE R D. Low complexity adaptive intra-refresh rate for real-time wireless video transmission[C]//Proceedings of the Signal and Information Processing Association Annual Summit and Conference (APSIPA). Piscataway: IEEE Press, 2014: 1-5.
[44]	ZHOU Y R, LI G Q, NING S S. A new feedback-based intra refresh method for robust video coding[C]//Proceedings of the 2015 International Conference on Computer Science and Applications (CSA). Piscataway: IEEE Press, 2015: 218-221.
[45]	HENDRY, WANG Y K, CHEN J L, et al. CE11/AHG14: Test 3.1-An approach for support of GRA/GDR[EB/OL]. [2020-07-04]. http://phenix.it-sudparis.eu/jvet/doc_end_user/documents/15_Gothenburg/wg11/JVET-O124-v3.zip.
[46]	LEE Y G, SONG B C. An intra-frame rate control algorithm for ultralow delay H. 264/advanced video coding (AVC)[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(5): 747-752. doi: 10.1109/TCSVT.2009.2017413
[47]	CHOI H, NAM J, YOO J, et al. Rate control based on unified RQ model for HEVC[EB/OL]. [2020-07-04]. http://phenix.int-evry.fr/jct/doc_end_user/documents/8_San%20Jose/wg11/JCTVC-H0213-v3.zip.
[48]	CHOI H, YOO J, NAM J, et al. Pixel-wise unified rate-quantization model for multi-level rate control[J]. IEEE Journal of Selected Topics in Signal Processing, 2013, 7(6): 1112-1123. doi: 10.1109/JSTSP.2013.2272241
[49]	WANG S, MA S, WANG S, et al. Quadratic ρ-domain based rate control algorithm for HEVC[C]//Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processin(ICASSP). Piscataway: IEEE Press, 2013: 1695-1699.
[50]	LI B, LI H, LI L, et al. λ domain rate control algorithm for high efficiency video coding[J]. IEEE Transactions on Image Processing, 2014, 23(9): 3841-3854. doi: 10.1109/TIP.2014.2336550
[51]	GAO W, KWONG S, JIA Y. Joint machine learning and game theory for rate control in high efficiency video coding[J]. IEEE Transactions on Image Processing, 2017, 26(12): 6074-6089. doi: 10.1109/TIP.2017.2745099
[52]	KWONG S, ZHOU M, XUEKAI W E I, et al. Rate control method based on deep reinforcement learning for dynamic video sequences in HEVC[J]. IEEE Transactions on Multimedia, 2020, 99: 1. http://www.researchgate.net/publication/341198848_Rate_Control_Method_Based_on_Deep_Reinforcement_Learning_for_Dynamic_Video_Sequences_in_HEVC
[53]	JIANG C, NOOSHABADI S. A scalable massively parallel motion and disparity estimation scheme for multiview video coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(2): 346-359. doi: 10.1109/TCSVT.2015.2402853
[54]	SHAHID M U, AHMED A, MARTINA M, et al. Parallel H. 264/AVC fast rate-distortion optimized motion estimation by using a graphics processing unit and dedicated hardware[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(4): 701-715. doi: 10.1109/TCSVT.2014.2351111
[55]	XIAO B, WANG H, WU J, et al. A multi-grained parallel solution for HEVC encoding on heterogeneous platforms[J]. IEEE Transactions on Multimedia, 2019, 21(12): 2997-3009. doi: 10.1109/TMM.2019.2916462
[56]	MOMCILOVIC S, ROMA N, SOUSA L. Exploiting task and data parallelism for advanced video coding on hybrid CPU + GPU platforms[J]. Journal of Real-Time Image Processing, 2016, 11(3): 571-587. doi: 10.1007/s11554-013-0357-y
[57]	INTEL. Scalable video technology for HEVC encoder (SVT-HEVC Encoder)[EB/OL]. (2020-08-20)[2020-08-26]. https://github.com/OpenVisualCloud/SVT-HEVC.
[58]	SURUR. TPCAST's wireless Adapter coming to the Oculus Rift next week[EB/OL]. (2017-11-08)[2020-08-26]. https://mspoweruser.com/tpcasts-wireless-adapter-coming-oculus-rift-next-week/.
[59]	BUYSSCHAERT C, DESCAMPE A, FÖßEL S, et al. Overview of JPEG XS[EB/OL]. 2018[2020-08-26]. https://jpeg.org/jpegxs/index.html.
[60]	AMIMON, Ltd. Wireless home digital interface^TM specification v1.0 Revision 33[EB/OL]. [2020-07-04]. https://www.amimon.com/proavl.
[61]	NETINT. Codensity T408 video transcoder product brief[EB/OL]. [2020-08-26]. https://www.netint.ca/product/t400_transcoder/.
[62]	INTEL. Scalable video technology for the visual cloud[EB/OL]. [2020-08-26]. https://01.org/sites/default/files/documentation/svt_aws_wp.pdf.
[63]	CAST. H264-E-BPF ultra-fast AVC/H. 264 baseline profile encoder[EB/OL]. [2020-08-26]. https://www.cast-inc.com/compression/avc-hevc-video-compression/h264-e-bpf/.
[64]	RICHTER T, KEINERT J, FOESSEL S, et al. JPEG-XS-A high-quality mezzanine image codec for video over IP[J]. SMPTE Motion Imaging Journal, 2018, 127(9): 39-49. doi: 10.5594/JMI.2018.2862098
[65]	REZNIC Z. Successive refinement video compression: USA, WO2018025211A1[P]. 2018-02-08.