基于编解码双路卷积神经网络的视觉自定位方法

贾瑞明; 刘圣杰; 李锦涛; 王赟豪; 潘海侠

doi:10.13700/j.bh.1001-5965.2019.0046

基于编解码双路卷积神经网络的视觉自定位方法

doi: 10.13700/j.bh.1001-5965.2019.0046

1.
北方工业大学信息学院, 北京 100144
2.
北京航空航天大学软件学院, 北京 100083

基金项目:

国家重点研发计划 2017YFB0802300

北京市教委面上项目 KM201510009005

北方工业大学学生科技活动项目 110051360007

详细信息

作者简介:
贾瑞明男, 博士, 助理研究员。主要研究方向:计算机视觉、深度学习、模式识别

刘圣杰男, 硕士研究生。主要研究方向:计算机视觉、深度学习

通讯作者:
贾瑞明, E-mail: jiaruiming@ncut.edu.cn

中图分类号: V249.32⁺9;TP391
计量
- 文章访问数: 827
- HTML全文浏览量: 87
- PDF下载量: 354
- 被引次数: 0
出版历程
- 收稿日期: 2019-02-13
- 录用日期: 2019-05-18
- 网络出版日期: 2019-10-20

A visual localization method based on encoder-decoder dual-stream CNN

1.
School of Information Science and Technology, North China University of Technology, Beijing 100144, China
2.
College of Software, Beihang University, Beijing 100083, China

Funds:

National Key R & D Program of China 2017YFB0802300

The General Program of Beijing Municipal Education Commission KM201510009005

Science and Technology Activities for Students of NCUT 110051360007

More Information

Corresponding author: JIA Ruiming, E-mail: jiaruiming@ncut.edu.cn

摘要

摘要:
为了从单张RGB图像估计出相机的位姿信息，提出了一种深度编解码双路卷积神经网络（CNN），提升了视觉自定位的精度。首先，使用编码器从输入图像中提取高维特征；然后，使用解码器提升特征的空间分辨率；最后，通过多尺度位姿预测器输出位姿参数。由于位置和姿态的特性不同，网络从解码器开始采用双路结构，对位置和姿态分别进行处理，并且在编解码之间增加跳跃连接以保持空间信息。实验结果表明：所提网络的精度与目前同类型算法相比有明显提升，其中相机姿态角度精度有较大提升。
- 视觉自定位 /
- 编解码结构 /
- 卷积神经网络(CNN) /
- 跳跃连接 /
- 双路网络
Abstract:
In order to calculate the camera pose from a single RGB image, a deep encoder-decoder dual-stream convolutional neural network (CNN) is proposed, which can improve the accuracy of visual localization. The network first uses an encoder to extract advanced features from input images. Second, the spacialresolution is enhancedby a pose decoder.Finally, a multi-scale estimator is used to output pose parameters. Becauseof the differentperformance of position and orientation, the network adopts a dual-stream structure from the decoder to process the position and orientationseparately. To restore the spatial information, several skip connections are added to encoder-decoder architecture. The experimental results show that the accuracy of the network is obviously improved compared with the congener state-of-the-art algorithms, and the orientation accuracy of camera pose is improved dramatically.
- visual localization /
- encoder-decoder architecture /
- convolutional neural network (CNN) /
- skip connection /
- dual-stream network

HTML全文

图 1 BiLocNet结构

Figure 1. Architecture of BiLocNet

下载: 全尺寸图片幻灯片

图 2 Inception-Resnet-V2编码器结构

Figure 2. Architecture of encoder Inception-Resnet-V2

下载: 全尺寸图片幻灯片

图 3 MSR模块结构

Figure 3. Module architecture of MSR

下载: 全尺寸图片幻灯片

图 4 SSR模块结构

Figure 4. Module architecture of SSR

下载: 全尺寸图片幻灯片

图 5 7-scenes数据集样例

Figure 5. Samples of 7-scenes dataset

下载: 全尺寸图片幻灯片

图 6 全连接预测器

Figure 6. FC estimator

下载: 全尺寸图片幻灯片

图 7 GAP预测器

Figure 7. GAP estimator

下载: 全尺寸图片幻灯片

图 8 有无预训练的BiLocNet损失函数曲线

Figure 8. Loss function curves of pre-trained and non-pre-trained BiLocNet

下载: 全尺寸图片幻灯片

表 1 不同场景下不同算法的位置误差和姿态误差

Table 1. Position error and orientation error with different scenes for various algorithms

场景	位置误差/m, 姿态误差/(°)
场景	PoseNet^[11]	Bayesian DS^[15]	LSTM-Pose^[13]	VidLoc^[12]	Hourglass^[14]	BiLocNet
Chess	0.32, 8.12	0.28, 7.05	0.24, 5.77	0.18, N/A	0.15, 6.17	0.13, 5.13
Fire	0.47, 14.4	0.43, 12.52	0.34, 11.9	0.26, N/A	0.27, 10.84	0.29, 10.48
Heads	0.29, 12.0	0.25, 12.72	0.21, 13.7	0.14, N/A	0.19, 11.63	0.16, 12.67
Office	0.48, 7.68	0.30, 8.92	0.30, 8.08	0.26, N/A	0.21, 8.48	0.25, 6.82
Pupkin	0.47, 8.42	0.36, 7.53	0.33, 7.00	0.36, N/A	0.25, 8.12	0.25, 5.23
Kitchen	0.59, 8.64	0.45, 9.80	0.37, 8.83	0.32, N/A	0.27, 10.15	0.26, 6.95
Stairs	0.47, 13.8	0.42, 13.06	0.40, 13.7	0.26, N/A	0.29, 12.46	0.33, 9.86
均值	0.44, 10.44	0.35, 10.22	0.31, 9.85	0.25, N/A	0.23, 9.69	0.23, 8.16

下载: 导出CSV

表 2 不同预测器的网络精度对比

Table 2. Comparison of network precision of different estimators

预测器	位置误差/m	姿态误差/(°)
全连接预测器	0.26	8.03
GAP预测器	0.14	5.21
多尺度位姿预测器	0.13	5.13

下载: 导出CSV

表 3 α权重损失函数与σ权重损失函数的网络精度对比

Table 3. Comparison of network precision between α weights loss function and σ weights loss function

损失函数	位置误差/m	姿态误差/(°)
α权重	0.19	6.55
σ权重	0.17	5.64

下载: 导出CSV

表 4 有无迁移学习的网络精度对比

Table 4. Comparison of network precision with and without transfer learning

有无迁移学习	位置误差/m	姿态误差/(°)
有	0.32	10.64
无	0.29	10.48

下载: 导出CSV

参考文献(26)

[1]	CHEN D M, BAATZ G, KOSER K, et al.City-scale landmark identification on mobile devices[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE Press, 2011: 12258110.
[2]	TORII A, SIVIC J, PAJDLA T, et al.Visual place recognition with repetitive structures[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE Press, 2013: 883-890.
[3]	SCHINDLER G, BROWN M, SZELISKI R.City-scale location recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2007: 1-7.
[4]	ARTH C, PIRCHHEIM C, VENTURA J, et al.Instant outdoor localization and SLAM initialization from 2.5 D maps[J].IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11):1309-1318. doi: 10.1109/TVCG.2015.2459772
[5]	POGLITSCH C, ARTH C, SCHMALSTIEG D, et al.A particle filter approach to outdoor localization using image-based rendering[C]//IEEE International Symposium on Mixed and Augmented Reality(ISMAR).Piscataway, NJ: IEEE Press, 2015: 132-135.
[6]	SATTLER T, LEIBE B, KOBBELT L.Improving image-based localization by active correspondence search[C]//Proceedings of European Conference on Computer Vision.Berlin: Springer, 2012: 752-765.
[7]	LI Y, SNAVELY N, HUTTENLOCHER D, et al.Worldwide pose estimation using 3D point clouds[C]//Proceedings of European Conference on Computer Vision.Berlin: Springer, 2012: 15-29.
[8]	CHOUDHARY S, NARAYANAN P J.Visibility probability structure from SFM datasets and applications[C]//Proceedings of European Conference on Computer Vision.Berlin: Springer, 2012: 130-143.
[9]	SVARM L, ENQVIST O, OSKARSSON M, et al.Accurate localization and pose estimation for large 3D models[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2014: 532-539.
[10]	SHOTTON J, GLOCKER B, ZACH C, et al.Scene coordinate regression forests for camera relocalization in RGB-D images[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2013: 2930-2937.
[11]	KENDALL A, GRIMES M, CIPOLLA R.PoseNet: A convolutional network for real-time 6-DOF camera relocalization[C]//Proceedings of IEEE International Conference on Computer Vision.Piscataway, NJ: IEEE Press, 2015: 2938-2946.
[12]	WALCH F, HAZIRBAS C, LEAL-TAIXÉ L, et al.Image-based localization with spatial LSTMs[EB/OL].(2016-11-23)[2018-12-25].https://arxiv.org/pdf/1611.07890v1.
[13]	CLARK R, WANG S, MARKHAM A, et al.VidLoc: A deep spatio-temporal model for 6-DOF video-clip relocalization[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2017: 6856-6864.
[14]	KENDALL A, CIPOLLA R.Modelling uncertainty in deep learning forcamera relocalization[C]//Proceedings of IEEE International Conference on Robotics and Automation (ICRA).Piscataway, NJ: IEEE Press, 2016: 4762-4769.
[15]	MELEKHOV I, YLIOINAS J, KANNALA J, et al.Image-based localization using hourglass networks[EB/OL].(2017-08-24)[2018-12-25].https://arxiv.org/abs/1703.07971.
[16]	LI R, LIU Q, GUI J, et al.Indoor relocalization in challenging environments with dual-stream convolutional neural networks[J].IEEE Transactions on Automation Science and Engineering, 2018, 15(2):651-662. doi: 10.1109/TASE.2017.2664920
[17]	RADWAN N, VALADA A, BURGARD W.Vlocnet++: Deep multitask learning for semantic visual localization and odometry[EB/OL].(2016-10-11)[2018-12-25].https://arxiv.org/abs/1804.08366.
[18]	BRAHMBHATT S, GU J, KIM K, et al.Geometry-aware learning of maps for camera localization[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2018: 2616-2625.
[19]	LI X, YLIOINAS J, KANNALA J.Full-frame scene coordinate regression for image-based localization[EB/OL].(2018-01-25)[2018-12-25].https://arxiv.org/abs/1802.03237.
[20]	LASKAR Z, MELEKHOV I, KALIA S, et al.Camera relocalization by computing pairwise relative poses using convolutional neural network[C]//Proceedings of IEEE International Conference on Computer Vision.Piscataway, NJ: IEEE Press, 2017: 929-938.
[21]	KENDALL A, GAL Y, CIPOLLA R.Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Piscataway, NJ: IEEE Press, 2018: 7482-7491.
[22]	SZEGEDY C, IOFFE S, VANHOUCKE V, et al.Inception-v4, inception-resnet and the impact of residual connections on learning[C]//Thirty-First AAAI Conference on Artificial Intelligence, 2017: 4-12.
[23]	KENDALL A, GAL Y.What uncertainties do we need in Bayesian deep learning for computer vision [EB/OL].(2017-10-05)[2018-12-26].https://arxiv.org/abs/1703.04977.
[24]	IZADI S, KIM D, HILLIGES O, et al.KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera[C]//Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology.New York: ACM, 2011: 559-568.
[25]	HE K M, ROSS G, PIOTR D.Rethinking imagenet pre-training[EB/OL].(2015-11-21)[2018-12-25].https://arxiv.org/abs/1811.08883.
[26]	WU Y X, HE K M.Group normalization[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018, 3-19.