基于眼动、位姿及场景视频的人体运动方向预测方法

doi:10.13700/j.bh.1001-5965.2020.0316

北京航空航天大学学报 ›› 2021, Vol. 47 ›› Issue (9): 1857-1865.doi: 10.13700/j.bh.1001-5965.2020.0316

基于眼动、位姿及场景视频的人体运动方向预测方法

张卿¹, 王兴坚^1,2, 苗忆南¹, 王少萍^1,2, Alexander I. GAVRILOV³

1. 北京航空航天大学自动化科学与电气工程学院, 北京 100083;
2. 北京航空航天大学大数据精准医疗高精尖创新中心, 北京 100083;
3. 莫斯科国立鲍曼技术大学自动控制系统系, 莫斯科 105005

收稿日期:2020-07-03 发布日期:2021-10-09
通讯作者: 王兴坚 E-mail:wangxj@buaa.edu.cn
基金资助:
国防基础科研项目（JCKY2018601C107）；国家自然科学基金（51675019，51620105010）

Human motion direction prediction method based on eye tracking, pose and scene video

ZHANG Qing¹, WANG Xingjian^1,2, MIAO Yinan¹, WANG Shaoping^1,2, Alexander I. GAVRILOV³

1. School of Automation Science and Electrical Engineering, Beihang University, Beijing 100083, China;
2. Advanced Innovation Center for Big Data-Based Precision Medicine, Beihang University, Beijing 100083, China;
3. Department of Automatic Control Systems, Bauman Moscow State Technical University, Moscow 105005, Russia

Received:2020-07-03 Published:2021-10-09
Supported by:
Defense Industrial Technology Development Program (JCKY2018601C107); National Natural Science Foundation of China (51675019,51620105010)

摘要/Abstract

摘要： 外骨骼机器人作为新提出的改善、提高人类生活能力的智能设备，同样需要高效智能的人机交互系统，而人机交互的第一步，则是精准预测人的行为意图。从外骨骼机器人的顶层控制角度出发，介绍了人体运动意图识别和外骨骼机器人智能交互能力的研究现状，并对人体运动方向识别进行了研究。提出了一种结合眼动信息、位姿信息及场景视频信息的多信息融合的人体运动意图识别网络架构，并进行了采集设备的穿戴实验。利用实验数据，对提出的网络方法进行了实验验证。结果表明：所提出的人体运动方向的识别系统，可以预测出人体运动过程中的运动方向。

关键词: 眼睛追踪, 预测方法, 运动方向, 人体行为, 人机交互

Abstract: Exoskeleton robots, as newly proposed smart devices to improve and enhance human life ability, require efficient and intelligent human-computer interaction systems, and the first step of human-computer interaction is to accurtely predict human behavior intention. From the perspective of top-level control of exoskeleton robots, the current states and progress of human motion intention recognition and the intelligent interaction capabilities of exoskeleton robots are reported. Then, the recognition of human motion direction is studied. A network framework of human motion intention recognition combining eye tracking information, position and posture information, and scene video information is proposed, and wearable experiments of acquisition devices are carried out. The predictive capability of the network has been proved by experiments. The results show that the proposed recognition system can predict the movement direction during human movement.

Key words: eye tracking, prediction method, direction of movement, human behavior, human computer interaction

中图分类号:

TP391.4

张卿, 王兴坚, 苗忆南, 王少萍, Alexander I. GAVRILOV. 基于眼动、位姿及场景视频的人体运动方向预测方法[J]. 北京航空航天大学学报, 2021, 47(9): 1857-1865.

ZHANG Qing, WANG Xingjian, MIAO Yinan, WANG Shaoping, Alexander I. GAVRILOV. Human motion direction prediction method based on eye tracking, pose and scene video[J]. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(9): 1857-1865.

参考文献

[1] JACOBSEN S C,OLIVIER M,SMITH F M,et al.Research robots for applications in artificial intelligence,teleoperation and entertainment[J].The International Journal of Robotics Research,2004,23(4-5):319-330.
[2] LIANG J W,JIANG L,NIEBLES J C,et al.Peeking into the future:Predicting future person activities and locations in videos[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Piscataway:IEEE Press,2019:2960-2963.
[3] 王昕.面向下肢康复机器人的运动意图识别技术研究[D].杭州:浙江大学,2019:97-98.WANG X.Research on motion intention recognition technology for lower limb rehabilitation robot[D].Hangzhou:Zhejiang University,2019:97-98(in Chinese).
[4] ALAHI A,GOEL K,RAMANATHAN V,et al.Social LSTM:Human trajectory prediction in crowded spaces[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway:IEEE Press,2016:961-971.
[5] BHATTACHARYYA A,FRITZ M,SCHIELE B.Long-term on-board prediction of people in traffic scenes under uncertainty[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workships (CVPRW).Piscataway:IEEE Press,2018:4194-4202.
[6] CHANDRA R,GUAN T R,PANUGANTI S,et al.Forecasting trajectory and behavior of road-agents using spectral clustering in graph-LSTMs[J].IEEE Robotics and Automation Letters,2020,5(3):4882-4890.
[7] CHANDRA R,BHATTACHARYA U,BERA A,et al.TraPHic:Trajectory prediction in dense and heterogeneous traffic using weighted interactions[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway:IEEE Press,2019:8475-8484.
[8] XU Y Y,PIAO Z X,GAO S H.Encoding crowd interaction with deep neural network for pedestrian trajectory prediction[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Piscataway:IEEE Press,2018:5275-5284.
[9] HAYASHI T,KAWAMOTO H,SANKAI Y.Control method of robot suit HAL working as operator's muscle using biological and dynamical information[C]//2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway:IEEE Press,2005:3063-3068.
[10] 张富平.图像去噪增强算法的研究[D].重庆:重庆大学,2014:1-2.ZHANG F P.Research of image enhancement and image denoising[D].Chongqing:Chongqing University,2014:1-2(in Chinese).
[11] 苏泫.基于IMU预积分的视觉惯性里程计系统[D].广州:华南理工大学,2018:5-7.SU X.Visual-inertial odometry based on IMU pre-integration[D].Guangzhou:South China University of Technology,2018:5-7(in Chinese).
[12] QIN T,LI P L,SHEN S J.VINS-Mono:A robust and versatile monocular visual-inertial state estimator[J].IEEE Transactions on Robotics,2018,34(4):1004-1020.
[13] 张裕天.基于视觉感知的多模态多任务端到端自动驾驶方法研究[D].广州:华南理工大学,2019:43-45.ZHANG Y T.Research on multi-modal multi-task end-to-end autonomous driving method based on visual perception[D].Guangzhou:South China University of Technology,2019:43-45(in Chinese).
[14] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway:IEEE Press,2016:770-778.
[15] FLETCHER D,IAN P.3D数学基础:图形与游戏开发[M].史银雪,陈洪,王荣静,译.北京:清华大学出版社,2005:154-157.FLETCHER D,IAN P.3D math primer for graphics and game development[M].SHI Y X,CHEN H,WANG R J,translated.Beijing:Tsinghua University Press,2005:154-157(in Chinese).
[16] FU R,ZHANG Z,LI L.Using LSTM and GRU neural network methods for traffic flow prediction[C]//2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC).Piscataway:IEEE Press,2016:324-328.
[17] KINGMA D P,BA J.Adam:A method for stochastic optimization[EB/OL].(2015-07-23)[2020-06-23].https://arxiv.org/abs/1412.6980.

基于眼动、位姿及场景视频的人体运动方向预测方法

Human motion direction prediction method based on eye tracking, pose and scene video

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

编辑推荐

Metrics

本文评价

[1]	燕学智, 李瑞格, 武秋爽. 表面触觉再现技术现状和评估方法[J]. 北京航空航天大学学报, 2018, 44(10): 2078-2095.
[2]	陈畅宇, 于勇, 赵罡. 基于数字样机的飞机研制项目监控技术[J]. 北京航空航天大学学报, 2014, 40(2): 198-203.
[3]	王衍洋, 曹义华. 民航安全指数结果分析与预测[J]. 北京航空航天大学学报, 2011, 37(10): 1223-1227.
[4]	赵罡, 王超, 侯文君, 金钺. 复杂产品虚拟装配系统的人机交互技术[J]. 北京航空航天大学学报, 2009, 35(2): 137-141.
[5]	巩应奎, 范玉青, 吴志方. 笔式三维建模系统的隐式映射机制[J]. 北京航空航天大学学报, 2006, 32(05): 603-606.
[6]	巩应奎, 范玉青, 吴志方. 笔式草图的增量识别[J]. 北京航空航天大学学报, 2006, 32(05): 621-624.
[7]	郝爱民, 张蓓, 赵沁平. 计算机虚拟北航的建造与人机交互的实现[J]. 北京航空航天大学学报, 2000, 26(4): 435-438.