基于Q-Learning算法和神经网络的飞艇控制

doi:10.13700/j.bh.1001-5965.2016.0903

北京航空航天大学学报 ›› 2017, Vol. 43 ›› Issue (12): 2431-2438.doi: 10.13700/j.bh.1001-5965.2016.0903

基于Q-Learning算法和神经网络的飞艇控制

聂春雨¹, 祝明¹, 郑泽伟², 武哲¹

1. 北京航空航天大学航空科学与工程学院, 北京 100083;
2. 北京航空航天大学自动化科学与电气工程学院, 北京 100083

收稿日期:2016-11-29 修回日期:2017-02-06 出版日期:2017-12-20 发布日期:2017-03-23
通讯作者: 祝明 E-mail:zhuming@buaa.edu.cn
作者简介:聂春雨,男,博士研究生。主要研究方向:机载软件技术、临近空间低速飞行器智能控制技术;祝明,男,博士,副教授,博士生导师。主要研究方向:飞行器总体设计、无人飞行器系统设计、无人飞行器控制、综合航电与机载软件技术。
基金资助:
国家自然科学基金（61503010）；中央高校基本科研业务费专项资金（YWF-14-RSC-103）

Airship control based on Q-Learning algorithm and neural network

NIE Chunyu¹, ZHU Ming¹, ZHENG Zewei², WU Zhe¹

1. School of Aeronautic Science and Engineering, Beijing University of Aeronautics and Astronautics, Beijing 100083, China;
2. School of Automation Science and Electrical Engineering, Beijing University of Aeronautics and Astronautics, Beijing 100083, China

Received:2016-11-29 Revised:2017-02-06 Online:2017-12-20 Published:2017-03-23
Supported by:
National Natural Science Foundation of China (61503010); the Fundamental Research Funds for the Central Universities (YWF-14-RSC-103)

摘要/Abstract

摘要： 针对现代飞艇控制中动力学模型不确定性带来的系统建模和参数辨识工作较为复杂的问题，提出了一种基于自适应建模和在线学习机制的控制策略。设计了一种在分析实际运动的基础上建立飞艇控制马尔可夫决策过程（MDP）模型的方法，具有自适应性。采用Q-Learning算法进行在线学习并利用小脑模型关节控制器（CMAC）神经网络对动作值函数进行泛化加速。对本文方法进行仿真并与经过参数整定的PID控制器对比，验证了该控制策略的有效性。结果表明，在线学习过程能够在数小时内收敛，通过自适应方法建立的MDP模型能够满足常见飞艇控制任务的需求。本文所提控制器能够获得与PID控制器精度相当且更为智能的控制效果。

关键词: 飞艇, 马尔可夫决策过程(MDP), 机器学习, Q-Learning, 小脑模型关节控制器(CMAC)

Abstract: An autonomous on-line learning control strategy based on adaptive modeling mechanism was proposed aimed at system modeling and parameter identification problems resulting from dynamic model uncertainties in modern airship control. An adaptive method to establish airship control Markov decision process (MDP) model was introduced on the foundation of analyzing airship's actual motion. On-line learning was carried out by Q-Learning algorithm, and cerebellar model articulation controller (CMAC) network was brought in for generalization of action value functions to accelerate algorithm convergence speed. Simulations of this autonomous on-line learning controller and comparisons with parameters turned PID controllers in normal control tasks were presented to demonstrate Q-Learning controller's effectiveness. The results show that the controller's on-line learning processes can converge in a few hours and the airship control MDP model established by the adaptive method satisfies the need of normal control tasks. The controller designed in this paper obtains similar precision as PID controllers and performs even more intelligently.

Key words: airship, Markov decision process (MDP), machine learning, Q-Learning, cerebellar model articulation controller (CMAC)

中图分类号:

V274
V249.22

聂春雨, 祝明, 郑泽伟, 武哲. 基于Q-Learning算法和神经网络的飞艇控制[J]. 北京航空航天大学学报, 2017, 43(12): 2431-2438.

NIE Chunyu, ZHU Ming, ZHENG Zewei, WU Zhe. Airship control based on Q-Learning algorithm and neural network[J]. JOURNAL OF BEIJING UNIVERSITY OF AERONAUTICS AND A, 2017, 43(12): 2431-2438.

参考文献

[1] PRENTICE B E,KNOTTS R.Cargo airships:International competition[J].Journal of Transportation Technologies,2014,4:187-195.
[2] 赵达,刘东旭,孙康文,等.平流层飞艇研制现状、技术难点及发展趋势[J].航空学报,2016,37(1):45-56.ZHAO D,LIU D X,SUN K W,et al.Research status,technical difficulties and development trend of stratospheric airship[J].Acta Aeronautica et Astronautica Sinica,2016,37(1):45-56(in Chinese).
[3] 郭虓.平流层浮空器轨迹优化研究[D].北京:北京航空航天大学,2013:29-36.GUO X.Trajectory optimization research for stratospheric aerostat[D].Beijing:Beihang University,2013:29-36(in Chinese).
[4] KHOURY G A.Airship technology[M].New York:Cambridge University Press,2012:34-40.
[5] YANG Y,WU J,ZHENG W.Positioning control for an autonomous airship[J].Journal of Aircraft,2016,53(6):1638-1646.
[6] ZHENG Z W,ZHU M,SHI D L,et al.Hovering control for a stratospheric airship in unknown wind:AIAA-2014-0973[R].Reston:AIAA,2014.
[7] ZHENG Z,LIU L,ZHU M.Integrated guidance and control path following and dynamic control allocation for a stratospheric airship with redundant control systems[J].Proceedings of the Institution of Mechanical Engineers,Part G:Journal of Aerospace Engineering,2016,230(10):1813-1826.
[8] YANG Y,YAN Y,ZHU Z,et al.Positioning control for an unmanned airship using sliding mode control based on fuzzy approximation[J].Proceedings of the Institution of Mechanical Engineers,Part G:Journal of Aerospace Engineering,2014,228(14):2627-2640.
[9] ABBEEL P,COATES A,QUIGLEY M,et al.An application of reinforcement learning to aerobatic helicopter flight[C]//Advances in Neural Information Processing Systems,2007:1-8.
[10] 徐昕.增强学习与近似动态规划[M].北京:科学出版社,2010:18-27.XU X.Reinforcement learning and approximate dynamic programing[M].Beijing:Science Press,2010:18-27(in Chinese).
[11] PEARRE B,BROWN T X.Model-free trajectory optimization for unmanned aircraft serving as data ferries for widespread sensors[J].Remote Sensing,2012,4(10):2971-3005.
[12] RAGI S,CHONG E K P.UAV path planning in a dynamic environment via partially observable Markov decision process[J].IEEE Transactions on Aerospace and Electronic Systems,2013,49(4):2397-2412.
[13] DUNN C,VALASEK J,KIRKPATRICK K.Unmanned air system search and localization guidance using reinforcement learning:AIAA-2012-2589[R].Reston:AIAA,2012.
[14] ZHANG B,MAO Z,LIU W,et al.Geometric reinforcement learning for path planning of UAVs[J].Journal of Intelligent & Robotic Systems,2015,77(2):391-409.
[15] FAUST A.Reinforcement learning and planning for preference balancing tasks[J].AI Matters,2015,1(3):8-12.
[16] KO J,KLEIN D J,FOX D,et al.Gaussian processes and reinforcement learning for identification and control of an autonomous blimp[C]//Proceedings 2007 IEEE International Conference on Robotics and Automation.Piscataway,NJ:IEEE Press,2007:742-747.
[17] ROTTMANN A,PLAGEMANN C,HILGERS P,et al.Autonomous blimp control using model-free reinforcement learning in a continuous state and action space[C]//2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway,NJ:IEEE Press,2007:1895-1900.
[18] LIN C M,PENG Y F.Adaptive CMAC-based supervisory control for uncertain nonlinear systems[J].IEEE Transactions on Systems,Man,and Cybernetics,Part B(Cybernetics),2004,34(2):1248-1260.
[19] SCHMIDT D K.Modeling and near-space station keeping control of a large high-altitude airship[J].Journal of Guidance,Control,and Dynamics,2007,30(2):540-547.
[20] LS-S1200 UAV airship system overview parameters[EB/OL].[2017-12-18].http://www.lonsan.com.cn/english/Products_1.asp?oneclass=5&pid=13.
[21] ATAEI M,YOUSEFI-KOMA A.Three-dimensional optimal path planning for waypoint guidance of an autonomous underwater vehicle[J].Robotics and Autonomous Systems,2015,67:23-32.

基于Q-Learning算法和神经网络的飞艇控制

Airship control based on Q-Learning algorithm and neural network

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	夏棒, EMILION Richard, 王惠文. Dirichlet混合样本的EM算法与动态聚类算法比较[J]. 北京航空航天大学学报, 2019, 45(9): 1805-1811.
[2]	张泰华, 姜鲁华, 周江华. 放飞过程中平流层飞艇运动与受力分析[J]. 北京航空航天大学学报, 2018, 44(4): 691-699.
[3]	游颖捷, 糜攀攀, 吕明云, 刘文定. 基于副气囊耦合的飞艇纵向运动影响分析[J]. 北京航空航天大学学报, 2016, 42(6): 1303-1310.
[4]	梁雅俊, 肖明清, 宋海方, 杨召, 梁鹏. 基于MDP的诊断策略构建方法[J]. 北京航空航天大学学报, 2016, 42(4): 844-850.
[5]	糜攀攀, 孟军辉, 吕明云. 浮升混合飞艇气动性能及总体参数分析[J]. 北京航空航天大学学报, 2015, 41(6): 1108-1116.
[6]	李可, 杨奕臻, 颜波. 基于学习的图像智能适配显示技术[J]. 北京航空航天大学学报, 2015, 41(6): 1147-1154.
[7]	肖厚地, 刘龙斌, 吕明云. X型尾翼临近空间飞艇隐身特性仿真[J]. 北京航空航天大学学报, 2015, 41(1): 181-186.
[8]	孟军辉, 张艳博, 吕明云. 平流层飞艇蒙皮材料织物纤维拔出过程分析[J]. 北京航空航天大学学报, 2014, 40(8): 1149-1153.
[9]	刘龙斌, 吕明云, 肖厚地, 曹帅. 基于压差梯度的平流层飞艇艇囊应力计算和仿真[J]. 北京航空航天大学学报, 2014, 40(10): 1386-1391.
[10]	梁浩全, 祝明, 姜光泰, 武哲. 基于改进CO-RS的平流层飞艇总体设计与优化[J]. 北京航空航天大学学报, 2013, 39(2): 239-243.
[11]	刘沛清, 马利川, 段中喆, 马蓉. 平流层飞艇螺旋桨相似准则分析与验证[J]. 北京航空航天大学学报, 2012, (7): 957-961.
[12]	肖治垣, 郦正能. 径向变体飞艇总体参数估算方法[J]. 北京航空航天大学学报, 2012, 38(5): 688-691,696.
[13]	郭虓, 祝明, 武哲. 综合热力学模型的平流层飞艇上升轨迹优化[J]. 北京航空航天大学学报, 2012, 38(10): 1346-1351.
[14]	宋静婧, 祝明, 武哲, 章磊. "人在回路"无人飞艇半实物仿真系统设计与实现[J]. 北京航空航天大学学报, 2011, 37(5): 595-599,609.
[15]	洪晔, 房建成. 基于HMDP的无人机三维路径规划[J]. 北京航空航天大学学报, 2009, 35(1): 100-103.