自调节图卷积UNet的三维人体姿态估计方法

马金林; 崔琦磊; 马自萍; 武江涛; 曹浩杰

doi:10.13700/j.bh.1001-5965.2022.0969

自调节图卷积UNet的三维人体姿态估计方法

doi: 10.13700/j.bh.1001-5965.2022.0969

马金林^{1, 2},
崔琦磊¹,
马自萍^3, ,,
武江涛¹,
曹浩杰¹

1.
北方民族大学计算机科学与工程学院，银川 750021
2.
图像图形智能信息处理国家民委重点实验室，银川 750021
3.
北方民族大学数学与信息科学学院，银川 750021

基金项目:

国家自然科学基金(62462001)；宁夏自然科学基金(2024AAC03147,2023AAC03264)；中央高校基本科研业务费专项资金(2023ZRLG02)

详细信息

通讯作者:
E-mail：2006041@nmu.edu.cn

中图分类号: V19；TP391.41
计量
- 文章访问数: 1102
- HTML全文浏览量: 327
- PDF下载量: 102
- 被引次数: 0
出版历程
- 收稿日期: 2022-12-06
- 录用日期: 2023-04-21
- 网络出版日期: 2023-05-25
- 整期出版日期: 2025-01-31

Self-adjusting graph convolution UNet method for 3D human pose estimation

MA Jinlin^{1, 2},
CUI Qilei¹,
MA Ziping^{3
, ,},
WU Jiangtao¹,
CAO Haojie¹

1.
College of Computer Science and Engineering，North Minzu University，Yinchuan 750021，China
2.
Key Laboratory of the National Ethnic Affairs Commission for Intelligent Processing of Image and Graphics，Yinchuan 750021，China
3.
College of Mathematics and Information Science，North Minzu University，Yinchuan 750021，China

Funds:

National Natural Science Foundation of China (62462001)；Natural Science Foundation of Ningxia (2024AAC03147, 2023AAC03264)；The Fundamental Research Funds for the Central Universities (2023ZRLG02)

More Information

Corresponding author: E-mail：2006041@nmu.edu.cn

摘要

摘要:
基于图卷积网络的三维人体姿态估计方法无法提取关节点的多尺度特征和未充分利用相邻节点的拓扑关系问题，提出自调节图卷积UNet的三维人体姿态估计方法M-Joint-UNet。M-Joint-UNet方法由Joint-UNet、自调节图卷积和融合损失3部分组成：Joint-UNet通过关节点池化与去池化改变特征图大小，以提取关节点的不同尺度特征；自调节图卷积通过可学习矩阵自动调节相邻节点或人体骨架结构的关系；使用L₁和L₂融合的损失缓解梯度爆炸。对比实验表明：所提方法在参数量和估计性能方面均获得了最优的结果，以Human3.6M的二维真实关节点作为输入的参数量仅为0.54×10⁶，MPJPE和P-MPJPE值分别为37.81 mm和30.21 mm。
- 三维人体姿态估计 /
- 图卷积 /
- Graph-UNet /
- 关节点池化 /
- 权重矩阵
Abstract:
A few issues with the graph convolutional network-based 3D human pose estimation technique include the shared weight of all nodes, the incapacity to extract the multi-scale properties of the nodes, and the failure to take advantage of the topological relationships between neighboring nodes. In this paper, self-adjusting graph convolution UNet method for 3D human pose estimation (M-Joint-UNet) is proposed. M-Joint-UNet is composed of three parts: Joint-UNet, self-adjusting graph convolution, and fusion loss. Joint-UNet changes the size of the feature map through node pooling and unpooling to extract features of different scales of nodes. Self-adjusting graph convolution uses L₁ and L₂ fusion to mitigate gradient explosions and automatically modifies the relationship between neighboring nodes or the human skeletal structure using a learnable matrix. Comparative experiments show that the proposed method obtains optimal results in terms of the number of parameters and the estimation performance. With the 2D ground truth of Human3.6M as the input, the number of parameters is only 0.54×10⁶, and the MPJPE and P-MPJPE values are 37.81 mm and 30.21 mm, respectively.
- 3D human pose estimation /
- graph convolution /
- Graph-UNet /
- joint pool /
- weight matrix

HTML全文

图 1 M-Joint-UNet网络结构

Figure 1. Network architecture of M-Joint-UNet

下载: 全尺寸图片幻灯片

图 2 关节点池化示意图

Figure 2. llustration of joint pool

下载: 全尺寸图片幻灯片

图 3 关节点去池化示意图

Figure 3. Illustration of joint unpool

下载: 全尺寸图片幻灯片

图 4 不同图卷积示意图

Figure 4. Illustration of different graph convolution

下载: 全尺寸图片幻灯片

图 5 不同个数自调节图卷积示意图

Figure 5. Illustration of self-adjusting graph convolution with different numbers

下载: 全尺寸图片幻灯片

图 6 M-Joint-UNet在Human3.6M数据集上的可视化展示

Figure 6. Visualization of M-Joint-UNet on Human3.6M dataset

下载: 全尺寸图片幻灯片

表 1 Joint-UNet各组件对姿态估计性能的影响

Table 1. Influence of Joint-UNet components on pose estimation performance

模型	通道数	参数量	MPJPE/mm	P-MPJPE/mm
PoseGCN	64	0.043×10⁶	81.02	68.56
PoseGCN w/JPool JUnpool	64	0.044×10⁶	59.13	48.14
Joint-UNet	64	0.056×10⁶	54.30	43.71

下载: 导出CSV

表 2 Joint-UNet中不同图卷积通道数的性能对比

Table 2. Performance comparison of graph convolution with different channel numbers in Joint-UNet

通道数	参数量	MPJPE/mm	P-MPJPE/mm
64	0.06×10⁶	54.30	43.71
128	0.21×10⁶	51.15	40.93
256	0.86×10⁶	49.78	40.27

下载: 导出CSV

表 3 Joint-UNet中使用不同图卷积的性能对比

Table 3. Performance comparison of different graph convolutions in Joint-UNet

模型	W的个数	通道数	参数量	MPJPE/mm	P-MPJPE/mm
普通图卷积	1	128	0.21×10⁶	51.15	40.93
自调节图卷积	1	128	0.22×10⁶	46.82	38.19

下载: 导出CSV

表 4 自调节图卷积使用不同个数权重矩阵的性能对比

Table 4. Performance comparison of self-adjusting graph convolution using different number of weight matrices

${\boldsymbol{W}}$的个数	是否权重共享	是否关节点分类	通道数	参数量	MPJPE/mm	P-MPJPE/mm
1	是	否	128	0.22×10⁶	46.82	38.19
2	否	是	128	0.31×10⁶	45.39	37.63
{16,8,4}	否	是	128	1.46×10⁶	45.08	37.39

下载: 导出CSV

表 5 M-Joint-UNet使用不同个数自调节图卷积的性能对比

Table 5. Performance comparison of M-Join-UNet using different numbers of self-adjusting graph convolution

自调节图卷积个数	通道数	参数量	MPJPE/mm	P-MPJPE/mm
1	128	0.31×10⁶	45.39	37.63
2	128	0.54×10⁶	38.76	31.73
3	128	0.65×10⁶	38.33	31.15

下载: 导出CSV

表 6 损失函数权重$\alpha $的选择

Table 6. Choice of weight $\alpha $ of loss function

$\alpha $	MPJPE/mm	P-MPJPE/mm
0	38.76	31.73
0.1	37.81	30.21
0.2	39.70	32.07
0.3	38.82	31.49
0.4	40.69	33.28
0.5	39.43	32.33

下载: 导出CSV

表 7 消融实验结果

Table 7. Ablation experiment results

模型	Joint-UNet	自调节图卷积	融合损失	通道数	参数量	MPJPE/mm	P-MPJPE/mm
Method-A	√			128	0.30×10⁶	44.67	37.53
Method-B		√		128	0.52×10⁶	43.41	35.63
Method-C	√		√	128	0.30×10⁶	43.86	36.48
Method-D		√	√	128	0.52×10⁶	42.37	34.13
Method-E	√	√		128	0.54×10⁶	38.76	31.73
M-Joint-UNet	√	√	√	128	0.54×10⁶	37.81	30.21

下载: 导出CSV

表 8 M-Joint-UNet与SOTA人体姿态估计模型的性能对比

Table 8. Performance comparison of M-Joint-UNet and SOTA human pose estimation models

模型	通道数	参数量	MPJPE/mm	P-MPJPE/mm
SemGCN^[31]	128	0.27×10⁶	42.14	33.53
SemGCN w/non-local^[31]	128	0.43×10⁶	40.78	31.46
High-order GCN^[19]	128	1.20×10⁶	39.68	31.69
HOIF-Net^[21]	64	0.54×10⁶	39.78	31.26
M-Joint-UNet	128	0.54×10⁶	37.81	30.21

下载: 导出CSV

表 9 prococal#1下M-Joint-UNet与其他基线模型的性能对比

Table 9. Performance comparison between M-Joint UNet and other baseline models under proccal # 1

模型	MPJPE/mm															平均 MPJPE/mm
模型	Direct	Discuss	Eating	Greet	Phone	Photo	Pose	Purch	Sitting	SittingD.	Smoke	Wait	WalkD.	Walk	WalkT.	平均 MPJPE/mm
文献[25]	51.8	56.2	58.1	59.0	69.5	78.4	55.2	58.1	74.0	94.6	62.3	59.1	65.1	49.5	52.4	62.9
文献[12]	52.8	54.8	54.2	54.3	61.8	67.2	53.1	53.6	71.7	86.7	61.5	53.4	61.6	47.1	53.4	59.1
文献[14]	51.5	58.9	50.4	57.0	62.1	65.4	49.8	52.7	69.2	85.2	57.4	58.4	43.6	60.1	47.7	58.0
文献[32]	50.1	54.3	57.0	57.1	66.6	73.3	53.4	55.7	72.8	88.6	60.3	57.7	62.7	47.5	50.6	60.5
文献[33]	48.5	54.4	54.4	52.0	59.4	65.3	49.9	52.9	65.8	71.1	56.6	52.9	60.9	44.7	47.8	55.8
文献[34]	48.6	54.5	54.2	55.7	62.2	72.0	50.5	54.3	70.0	78.3	58.1	55.4	61.4	45.2	49.7	58.0
文献[31]	47.3	60.7	51.4	60.5	61.1	49.9	47.3	68.1	86.2	55.0	67.8	61.0	42.1	60.6	45.3	57.6
文献[19]	49.0	54.5	52.3	53.6	59.2	71.6	49.6	49.8	66.0	75.5	55.1	53.8	58.5	40.9	45.4	55.7
文献[35]	47.1	52.8	54.2	54.9	63.8	72.5	51.7	54.3	70.9	85.0	58.7	54.9	59.7	43.8	47.1	58.1
文献[20]	51.0	55.3	54.0	54.6	62.4	76.0	51.6	52.7	79.3	87.1	58.4	56.0	61.8	48.1	44.1	59.5
文献[21]	47.0	53.7	50.9	52.4	57.8	71.3	50.2	49.1	63.5	76.3	54.1	51.6	56.5	41.7	45.3	54.8
本文	45.9	50.2	48.5	51.2	53.0	62.2	47.8	47.5	59.4	69.3	51.3	48.7	54.5	38.4	42.1	51.3

下载: 导出CSV

表 10 prococal#2下M-Joint-UNet与其他基线模型的性能对比

Table 10. Performance comparison between M-Joint-UNet and other baseline models under proccal # 2

模型	P-MPJPE/mm															平均 P-MPJPE/mm
模型	Direct	Discuss	Eating	Greet	Phone	Photo	Pose	Purch	Sitting	SittingD.	Smoke	Wait	WalkD.	Walk	WalkT.	平均 P-MPJPE/mm
文献[25]	39.5	43.2	46.4	47.0	51.0	56.0	41.4	40.6	56.5	69.4	49.2	45.0	49.5	38.0	43.1	47.7
文献[12]	42.1	44.3	45.0	45.4	51.5	53.0	43.2	41.3	59.3	73.3	51.0	44.0	48.0	38.3	44.8	48.3
文献[32]	38.2	41.7	39.6	45.2	45.8	46.5	37.8	42.7	52.4	62.9	45.3	40.9	45.3	38.6	38.4	44.1
文献[33]	47.5	50.5	48.3	49.3	50.7	55.2	46.1	48.0	61.1	78.1	51.1	48.3	52.9	41.5	46.4	51.7
文献[19]	38.6	42.8	41.8	43.4	44.6	52.9	37.5	38.6	53.3	60.0	44.4	40.9	46.9	32.2	37.9	43.7
文献[35]	36.7	39.5	41.5	42.6	46.9	53.5	38.2	36.5	52.1	61.5	45.0	42.7	45.2	35.3	40.2	43.8
文献[20]	38.4	43.1	42.9	44.0	47.8	56.0	39.3	39.8	61.8	67.1	46.1	43.4	48.4	40.7	35.1	46.3
文献[21]	36.9	42.1	40.3	42.1	43.7	52.7	37.9	37.7	51.5	60.3	43.9	39.4	45.4	31.9	37.8	42.9
本文	38.2	40.0	40.1	42.8	41.3	48.3	37.0	37.4	48.2	56.0	42.1	38.2	44.6	31.4	36.2	41.5

下载: 导出CSV

表 11 在MPI-INF-3DHP测试集下M-Joint-UNet与其他基线模型的性能对比

Table 11. Performance comparison of M-Joint-UNet with other baseline models in MPI-INF-3DHP test set

模型	训练集	PCK	AUC
文献[25]	Human3.6M	42.5	17.0
文献[29]	Human3.6M	64.7	31.7
文献[15]	Human3.6M+MPI-INF-3DHP	69.2	32.5
文献[32]	Human3.6M+MPI-INF-3DHP	69.0	32.0
文献[36]	Human3.6M	65.6	33.2
文献[33]	Human3.6M+MPI-INF-3DHP	71.9	35.3
文献[37]	Human3.6M	77.6	43.8
文献[21]	Human3.6M	72.8	36.5
文献[38]	Human3.6M	76.1	40.6
文献[39]	Human3.6M	77.0
文献[40]	Human3.6M	73.4
本文	Human3.6M	78.9	44.3

下载: 导出CSV

参考文献(40)

[1]	WENG J W, WENG C Q, YUAN J S. Spatio-temporal naive-Bayes nearest-neighbor (ST-NBNN) for skeleton-based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 445-454.
[2]	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 7444-7452.
[3]	LI M S, CHEN S H, CHEN X, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2019: 3590-3598.
[4]	JIANG S Y, SUN B, WANG L C, et al. Skeleton aware multi-modal sign language recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE Press, 2021: 3408-3418.
[5]	GAN J, WANG W Q. In-air handwritten English word recognition using attention recurrent translator[J]. Neural Computing and Applications, 2019, 31(7): 3155-3172.
[6]	LU D, LUO L. FMKit: An in-air-handwriting analysis library and data repository[C]//Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality. [S. l. ]: NSF PAR, 2020.
[7]	HASSANIN M, KHAN S, TAHTALI M. Visual affordance and function understanding[J]. ACM Computing Surveys, 2022, 54(3): 1-35.
[8]	GARCIA-SALGUERO M, GONZALEZ-JIMENEZ J, MORENO F A. Human 3D pose estimation with a tilting camera for social mobile robot interaction[J]. Sensors, 2019, 19(22): 4943.
[9]	GUI L Y, ZHANG K, WANG Y X, et al. Teaching robots to predict human motion[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2018: 562-567.
[10]	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[11]	PAVLAKOS G, ZHOU X W, DERPANIS K G, et al. Coarse-to-fine volumetric prediction for single-image 3D human pose[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 1263-1272.
[12]	SUN X, SHANG J X, LIANG S, et al. Compositional human pose regression[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2017: 2621-2630.
[13]	SUN X, XIAO B, WEI F, et al. Integral human pose regression[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2018: 536-553.
[14]	YANG W, OUYANG W L, WANG X L, et al. 3D human pose estimation in the wild by adversarial learning[C]//Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 5255-5264.
[15]	ZHOU X Y, HUANG Q X, SUN X, et al. Towards 3D human pose estimation in the wild: A weakly-supervised approach[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2017: 398-407.
[16]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[17]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[18]	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2017-02-22)[2022-12-01]. https://arxiv.org/abs/1609.02907.
[19]	ZOU Z, LIU K, WANG L, et al. High-order graph convolutional networks for 3D human pose estimation[C]//Proceedings of the 31st British Machine Vision Conference. [S. l. ]: BMVC, 2020: 1-13.
[20]	BANIK S, GARCÍA A M, KNOLL A. 3D human pose regression using graph convolutional network[C]//Proceedings of the IEEE International Conference on Image Processing. Piscataway: IEEE Press, 2021: 924-928.
[21]	QUAN J N, BEN HAMZA A. Higher-order implicit fairing networks for 3D human pose estimation[EB/OL]. (2021-11-01)[2022-12-01]. https://arxiv.org/abs/2111.00950.
[22]	RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional networks for biomedical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2015: 234-241.
[23]	GAO H Y, JI S W. Graph U-Nets[C]//Proceedings of the International Conference on Machine Learning. San Diego: PMLR, 2019: 2083-2092.
[24]	LIU K K, DING R Q, ZOU Z M, et al. A comprehensive study of weight sharing in graph networks for 3d human pose estimation[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2020: 318-334.
[25]	MARTINEZ J, HOSSAIN R, ROMERO J, et al. A simple yet effective baseline for 3d human pose estimation[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway: IEEE Press, 2017: 2659-2668.
[26]	IOFFE S, SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the International Conference on Machine Learning. San Diego: PMLR, 2015: 448-456.
[27]	NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proceedings of the 27th International Conference on Machine Learning. New York: ACM, 2010: 807-814.
[28]	IONESCU C, PAPAVA D, OLARU V, et al. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1325-1339.
[29]	MEHTA D, RHODIN H, CASAS D, et al. Monocular 3D human pose estimation in the wild using improved CNN supervision[C]// Proceedings of the International Conference on 3D Vision. Piscataway: IEEE Press, 2017: 506-516.
[30]	KINGMA D, BA J. Adam: A method for stochastic optimization[C]//Proceedings of the International Conference on Learning Representations. Appleton: ICLR, 2015: 1-15.
[31]	ZHAO L, PENG X, TIAN Y, et al. Semantic graph convolutional networks for 3D human pose regression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2019: 3420-3430.
[32]	FANG H S, XU Y L, WANG W G, et al. Learning pose grammar to encode human body configuration for 3D pose estimation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018, 6821-6828.
[33]	PAVLAKOS G, ZHOU X W, DANIILIDIS K. Ordinal depth supervision for 3D human pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2018: 7307-7316.
[34]	SHARMA S, VARIGONDA P T, BINDAL P, et al. Monocular 3D human pose estimation by generation and ordinal ranking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE Press, 2019: 2325-2334.
[35]	XU Y L, WANG W G, LIU T Y, et al. Monocular 3D pose estimation via pose grammar and data augmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6327-6344.
[36]	LUO C X, CHU X, YUILLE A. OriNet: A fully convolutional network for 3D human pose estimation[EB/OL]. (2018-11-12)[2022-12-01]. https://arxiv.org/abs/1811.04989.
[37]	ZENG A L, SUN X, HUANG F Y, et al. SRNet: Improving generalization in 3D human pose estimation with a split-and-recombine approach[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin: Springer, 2020: 507-523.
[38]	WANG J, HUANG S L, WANG X C, et al. PONet: Robust 3D human pose estimation via learning orientations only[EB/OL]. (2021-12-21)[2022-12-01]. https://arxiv.org/abs/2112.11153.
[39]	WANDT B, RUDOLPH M, ZELL P, et al. CanonPose: Self-supervised monocular 3D human pose estimation in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2021: 13289-13299.
[40]	ROY S K, CITRARO L, HONARI S, et al. On triangulation as a form of self-supervision for 3D human pose estimation[C]//Proceedings of the International Conference on 3D Vision. Piscataway: IEEE Press, 2022: 1-10.