基于三元组网络的单图三维模型检索

doi:10.13700/j.bh.1001-5965.2020.0057

北京航空航天大学学报 ›› 2020, Vol. 46 ›› Issue (9): 1691-1700.doi: 10.13700/j.bh.1001-5965.2020.0057

基于三元组网络的单图三维模型检索

杜雨佳^1,2,3, 李海生^1,2,3, 姚春莲^1,2,3, 蔡强^1,2,3

1. 北京工商大学计算机与信息工程学院, 北京 100048;
2. 农产品质量安全追溯技术及应用国家工程实验室, 北京 100048;
3. 食品安全大数据技术北京市重点实验室, 北京 100048

收稿日期:2020-02-28 发布日期:2020-09-22
通讯作者: 李海生 E-mail:lihsh@btbu.edu.cn
作者简介:杜雨佳女,硕士研究生。主要研究方向:计算机图形学;李海生男,博士,教授,博士生导师。主要研究方向:计算机图形学;姚春莲女,博士,副教授。主要研究方向:视频、图像处理、嵌入式系统设计;蔡强男,博士,教授,博士生导师。主要研究方向:计算机图形学。
基金资助:
国家自然科学基金（61877002）；北京市自然科学基金-丰台轨道交通前沿研究联合基金（L191009）；北京市教委科研团队建设项目（PXM2019_014213_000007）

Monocular image based 3D model retrieval using triplet network

DU Yujia^1,2,3, LI Haisheng^1,2,3, YAO Chunlian^1,2,3, CAI Qiang^1,2,3

1. School of Computer and Information Engineering, Beijing Technology and Business University, Beijing 100048, China;
2. National Engineering Laboratory For Agri-product Quality Traceability, Beijing 100048, China;
3. Beijing Key Laboratory of Big Data Technology for Food Safety, Beijing 100048, China

Received:2020-02-28 Published:2020-09-22
Supported by:
National Natural Science Foundation of China (61877002); Beijing Natural Science Foundation and Fengtai Rail Transit Frontier Research Joint Fund (L191009); Beijing Municipal Education Commission Research Team Construction Project(PXM2019_014213_000007)

摘要/Abstract

摘要： 随着媒体数据的多样化发展，联合图像与三维模型的跨域检索成为三维模型检索问题的一个新挑战。针对图像与三维模型差异大、难匹配问题，提出了一种基于三元组网络的跨域数据检索方法。以端到端的方式构建真实图像与三维模型的特征联合嵌入空间，通过特征间的距离度量不同模态数据之间的相似性，实现从单张图像检索相似的三维模型。为了提高跨域检索准确度，将三维模型用一组顺序视图表示，结合门控循环单元（GRU）聚合视图级特征，同时引入注意力机制提取图像特征，缩小真实图像与投影视图间的语义差异。实验结果表明：相比于同类方法，所提方法在两个跨域数据集上的检索平均准确率至少提升2.98%~3.05%。

关键词: 三维模型检索, 深度学习, 跨域检索, 三元组网络, 门控循环单元(GRU), 注意力机制

Abstract: With the diversified development of media data, the cross-domain retrieval between images and 3D models becomes a new challenge for 3D model retrieval. In view that images and 3D models are extremely different and hard to match, a cross-domain retrieval algorithm based on triple network is proposed to construct a joint embedding space for real images and 3D shapes in an end-to-end manner. Then the similarity between different modal data could be effectively computed by the distance in the space, leading to accurate retrieval of similar 3D models from single image. In order to improve the accuracy of cross-domain retrieval, the 3D model was represented by a set of sequential views, and the Gate Recurrent Unit (GRU) was utilized for view-level features to generate the global feature. In addition, an attention mechanism was introduced to extract image features and bridge the semantic gaps between the real image and the rendered 3D views. Experimental results show that the mean average precision can be improved by at least 2.98%-3.05% on two cross-domain datasets compared with other similar algorithms.

Key words: 3D model retrieval, deep learning, cross-domain retrieval, triplet network, Gate Recurrent Unit (GRU), attention mechanism

中图分类号:

TP183
TP391

杜雨佳, 李海生, 姚春莲, 蔡强. 基于三元组网络的单图三维模型检索[J]. 北京航空航天大学学报, 2020, 46(9): 1691-1700.

DU Yujia, LI Haisheng, YAO Chunlian, CAI Qiang. Monocular image based 3D model retrieval using triplet network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020, 46(9): 1691-1700.

参考文献

[1] BU S H,WANG L,HAN P C,et al.3D shape recognition and retrieval based on multi-modality deep learning[J].Neurocomputing,2017,259:183-193.
[2] 蔡轶珩,王雪艳,胡绍斌,等.基于多源图像弱监督学习的3D人体姿态估计[J].北京航空航天大学学报,2019,45(12):2375-2384.CAI Y H,WANG X Y,HU S B,et al.Three-dimensional human pose estimation based on multi-source image weakly-supervised learning[J].Journal of Beijing University of Aeronautics and Astronautics,2019,45(12):2375-2384(in Chinese).
[3] GIRDHAR R,FOUHEY D F,RODRIGUEZ M,et al.Learning a predictable and generative vector representation for objects[C]//European Conference on Computer Vision.Berlin:Springer,2016:484-499.
[4] TULSIANI S,GUPTA S,FOUHEY D F,et al.Factoring shape,pose,and layout from the 2d image of a 3d scene[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2018:302-310.
[5] IYER N,JAYANTI S,LOU K,et al.Three-dimensional shape searching:State-of-the-art review and future trends[J].Computer-Aided Design,2005,37(5):509-530.
[6] XIE J,FANG Y,ZHU F,et al.Deepshape:Deep learned shape descriptor for 3d shape matching and retrieval[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2015:1275-1283.
[7] MIAN A S,BENNAMOUN M,OWENS R A.Matching tensors for pose invariant automatic 3D face recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2005:120.
[8] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press,2012:1097-1105.
[9] 杨思晨,王华锋,王月海,等.深度学习机制与小波融合的超分辨率重建算法[J].北京航空航天大学学报,2020,46(1):189-197.YANG S C,WANG H F,WANG Y H,et al.Super-resolution reconstructing algorithm based on deep learning mechanism and wavelet fusion[J].Journal of Beijing University of Aeronautics and Astronautics,2020,46(1):189-197(in Chinese).
[10] GRABNER A,ROTH P M,LEPETIT V.Location field descriptors:Single image 3D model retrieval in the wild[C]//Proceedings of the 2019 International Conference on 3D Vision (3DV).Piscataway:IEEE Press,2019:583-593.
[11] WU Z Z,ZHANG Y H,ZENG M,et al.Joint analysis of shapes and images via deep domain adaptation[J].Computers & Graphics,2018,70:140-147.
[12] WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional block attention module[C]//European Conference on Computer Vision.Berlin:Springer,2018:3-19.
[13] CHO K,VAN MERRIËNBOER B,GULCEHRE C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL].(2014-06-03)[2020-02-25].https://arxiv.org/abs/1406.1078.
[14] LEE T,LIN Y L,CHIANG H Y,et al.Cross-domain image-based 3D shape retrieval by view sequence learning[C]//Proceedings of the 2018 International Conference on 3D Vision (3DV).Piscataway:IEEE Press,2018:258-266.
[15] LI W,LIU A,NIE W Z,et al.SHREC 2019-Monocular image based 3D model retrieval[EB/OL].(2019-01-28)[2020-02-25].https://www.iti-tju.org/MI3DOR19/.
[16] FANG Y,XIE J,DAI G,et al.3D deep shape descriptor[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2015:2319-2328.
[17] 李海生,武玉娟,郑艳萍,等.基于深度学习的三维数据分析理解方法研究综述[J/OL].计算机学报,2019,42:1-25.(2019-07-09)[2020-02-21].http://kns.cnki.net/kcms/detail/11.1826.TP.20190709.1509.002.html.LI H S,WU Y J,ZHENG Y P,et al.A survey of 3D data analysis and understanding based on deep learning[J/OL].Chinese Journal of Computers,2019,42:1-25.(2019-07-09)[2020-02-21].http://kns.cnki.net/kcms/detail/11.1826.TP.20190709.1509.002.html(in Chinese).
[18] OSADA R,FUNKHOUSER T,CHAZELLE B,et al.Shape distributions[J].ACM Transactions on Graphics (TOG),2002,21(4):807-832.
[19] MAHMOUDI M,SAPIRO G.Three-dimensional point cloud recognition via distributions of geometric distances[J].Graphical Models,2009,71(1):22-31.
[20] SUN J,OVSJANIKOV M,GUIBAS L.A concise and provably informative multi-scale signature based on heat diffusion[J].Computer Graphics Forum,2009,28(5):1383-1392.
[21] AUBRY M,SCHLICKEWEI U,CREMERS D.The wave kernel signature:A quantum mechanical approach to shape analysis[C]//Proceedings of 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).Piscataway:IEEE Press,2011:1626-1633.
[22] WANG P S,SUN C Y,LIU Y,et al.Adaptive O-CNN:A patch-based deep representation of 3D shapes[J].ACM Transactions on Graphics (TOG),2018,37(6):1-11.
[23] FENG Y,FENG Y,YOU H,et al.MeshNet:Mesh neural network for 3D shape representation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2019,33:8279-8286.
[24] QI C R,YI L,SU H,et al.Pointnet++:Deep hierarchical feature learning on point sets in a metric space[C]//Advances in Neural Information Processing Systems.Cambridge:MIT Press,2017:5099-5108.
[25] HUANG H B,KALOGERAKIS E,CHAUDHURI S,et al.Learning local shape descriptors from part correspondences with multiview convolutional networks[J].ACM Transactions on Graphics (TOG),2017,37(1):1-14.
[26] WANG P S,LIU Y,GUO Y X,et al.O-CNN:Octree-based convolutional neural networks for 3d shape analysis[J].ACM Transactions on Graphics (TOG),2017,36(4):1-11.
[27] HAN Z,SHANG M,LIU Z,et al.SeqViews2SeqLabels:Learning 3D global features via aggregating sequential views by RNN with attention[J].IEEE Transactions on Image Processing,2018,28(2):658-672.
[28] LAN S Y,YU R C,YU G,et al.Modeling local geometric structure of 3D point clouds using Geo-CNN[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2019:998-1008.
[29] SU H,MAJI S,KALOGERAKIS E,et al.Multi-view convolutional neural networks for 3d shape recognition[C]//Proceedings of the IEEE International Conference on Computer Vision.Piscataway:IEEE Press,2015:945-953.
[30] FENG Y,ZHANG Z,ZHAO X,et al.GVCNN:Group-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2018:264-272.
[31] JIANG J,BAO D,CHEN Z,et al.MLVCNN:Multi-loop-view convolutional neural network for 3D shape retrieval[C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2019,33:8513-8520.
[32] TASSE F P,DODGSON N.Shape2Vec:Semantic-based descriptors for 3D shapes,sketches and images[J].ACM Transactions on Graphics (TOG),2016,35(6):1-12.
[33] AUBRY M,MATURANA D,EFROS A A,et al.Seeing 3d chairs:Exemplar part-based 2d-3d alignment using a large dataset of cad models[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2014:3762-3769.
[34] MOTTAGHI R,XIANG Y,SAVARESE S.A coarse-to-fine model for 3d pose estimation and sub-category recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2015:418-426.
[35] KULIS B.Metric learning:A survey[J].Foundations and Trendsin Machine Learning,2013,5(4):287-364.
[36] XIANG Y,KIM W,CHEN W,et al.ObjectNet3D:A large scale database for 3d object recognition[C]//European Conference on Computer Vision.Berlin:Springer,2016:160-176.
[37] WANG F,KANG L,LI Y.Sketch-based 3d shape retrieval using convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2015:1875-1883.
[38] LI Y Y,SU H,QI C R,et al.Joint embeddings of shapes and images via CNN image purification[J].ACM Transactions on Graphics (TOG),2015,34(6):1-12.
[39] DAI G,XIE J,ZHU F,et al.Deep correlated metric learning for sketch-based 3d shape retrieval[C]//Proceedings of the AAAI Conference on Artificial Intelligence.Palo Alto:AAAI Press,2017:4002-4008.
[40] DAI G X,XIE J,FANG Y.Deep correlated holistic metric learning for sketch-based 3d shape retrieval[J].IEEE Transactions on Image Processing,2018,27(7):3374-3386.
[41] SCHROFF F,KALENICHENKO D,PHILBIN J.FaceNet:Aunified embedding for face recognition and clustering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Piscataway:IEEE Press,2015:815-823.
[42] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].(2014-09-04)[2020-02-25].https://arxiv.org/abs/1409.1556.

基于三元组网络的单图三维模型检索

Monocular image based 3D model retrieval using triplet network

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	陈猛夫. 基于迁移学习的暴恐图像自动识别[J]. 北京航空航天大学学报, 2020, 46(9): 1677-1681.
[2]	陶一宁, 苏峰, 袁培江, 王田苗, 钟涛, 郝静如. 联合足迹识别与监控视频分析的智能刑侦系统[J]. 北京航空航天大学学报, 2020, 46(9): 1730-1738.
[3]	张洁茹, 苏峰, 袁培江, 王田苗, 陶一宁, 丁东. 双光谱智能体温检测与健康大数据管理系统[J]. 北京航空航天大学学报, 2020, 46(9): 1739-1746.
[4]	张子昊, 周千里, 王蓉. 基于空间注意力机制的行人再识别方法[J]. 北京航空航天大学学报, 2020, 46(9): 1747-1755.
[5]	张子昊, 王蓉. 基于MobileFaceNet网络改进的人脸识别方法[J]. 北京航空航天大学学报, 2020, 46(9): 1756-1762.
[6]	邱博, 刘翔, 石蕴玉, 尚岩峰. 一种轻量化的多目标实时检测模型[J]. 北京航空航天大学学报, 2020, 46(9): 1778-1785.
[7]	陆展鸿, 单鲁斌, 苏立循, 焦雨欣, 王家骅, 王海霞. 基于U-Net的掌纹图像增强与ROI提取[J]. 北京航空航天大学学报, 2020, 46(9): 1807-1816.
[8]	李欣致, 董胜波, 崔向阳, 刘志哲, 郭广浩. 应用于智能芯片的可视化反馈系统研究[J]. 北京航空航天大学学报, 2020, 46(8): 1494-1502.
[9]	张阳, 牛之贤, 牛保宁, 常艳. 基于高分辨率网络的单声道歌声分离[J]. 北京航空航天大学学报, 2020, 46(8): 1555-1563.
[10]	李承昊, 茹乐, 何林远, 迟文升. 一种可变锚框候选区域网络的目标检测方法[J]. 北京航空航天大学学报, 2020, 46(8): 1610-1617.
[11]	吕娜, 周家欣, 陈卓, 刘鹏飞, 高维廷. 一种鲁棒性增强的机载网络流量分类方法[J]. 北京航空航天大学学报, 2020, 46(7): 1237-1246.
[12]	郑磊, 胡维多, 刘畅. 基于深度学习的大型陨石坑识别方法研究[J]. 北京航空航天大学学报, 2020, 46(5): 994-1004.
[13]	边亮, 罗霄阳, 李硕. 基于深度学习的图像拼接篡改检测[J]. 北京航空航天大学学报, 2020, 46(5): 1039-1044.
[14]	杨思晨, 王华锋, 王月海, 李锦涛, 王赟豪. 深度学习机制与小波融合的超分辨率重建算法[J]. 北京航空航天大学学报, 2020, 46(1): 189-197.
[15]	张克明, 蔡远文, 任元. 基于生成对抗网络的航天异常事件检测方法[J]. 北京航空航天大学学报, 2019, 45(7): 1329-1336.