Volume 52 Issue 11
Nov.  2026
Turn off MathJax
Article Contents
Yuan J B,Liu H Y,Ma B D,et al. UAV online path planning based on POMDP and parameterized MPC in uncertain environments[J]. Journal of Beijing University of Aeronautics and Astronautics,2026,52(11):3941-3957 (in Chinese)
Citation: Yuan J B,Liu H Y,Ma B D,et al. UAV online path planning based on POMDP and parameterized MPC in uncertain environments[J]. Journal of Beijing University of Aeronautics and Astronautics,2026,52(11):3941-3957 (in Chinese)

UAV online path planning based on POMDP and parameterized MPC in uncertain environments

doi: 10.13700/j.bh.1001-5965.2024.0662
Funds:

National Natural Science Foundation of China (52072309,12102345); Key Research and Development Program of Shaanxi Province (2019ZDLGY14-02-01,2022FP-40); Shenzhen Fundamental Research Program (JCYJ20190806152203506); Aeronautical Science Foundation (ASFC-2018ZC53026); Suzhou Municipal Science and Technology Bureau under Grant (ZXL2023177); Taicang Basic Research Program under Grant (TC2023JC13)

More Information
  • Corresponding author: E-mail:liuzhenbao@nwpu.edu.cn
  • Received Date: 06 Sep 2024
  • Accepted Date: 15 Nov 2024
  • Available Online: 16 Nov 2026
  • Publish Date: 22 Nov 2024
  • It is challenging to eliminate positioning and perception ambiguities when unmanned aerial vehicles (UAVs) operate autonomously in unfamiliar areas. In view of this, it is necessary to conduct research on path planning to enhance the effectiveness of the path under the interference of uncertain factors and thereby improve the target tracking ability of UAVs. The goal of applying model predictive control(MPC)to the real-time path planning system is currently accomplished by Tube-based-MPC, which supports the separation architecture of planning and control, and nominal belief state optimization (NBO), which offers a theoretical foundation for the uncertainty of target tracking. On this premise, this paper proposes an online path planning framework to solve the problems of target tracking and obstacle avoidance in unknown environments. First, the path planning problem is abstracted into a partially observable Markov decision process (POMDP) model, and the Kalman filter is used to estimate the belief state, integrating the uncertainty measure of target positioning into the framework. Secondly, based on the simplified 3-dimensional motion model, a tightened constraint Tube-based-MPC optimization framework is constructed to solve the POMDP. The path planning for a specific continuous time span in the future is achieved by building a nonlinear optimization problem that includes the belief state and unknown barriers. Then, the shapes of UAVs and obstacles are fully considered in the optimization problem, and a full-dimensional obstacle avoidance distance metric is introduced to significantly expand the feasible region of the optimization problem. Finally, the Q-learning reinforcement learning framework is introduced, and MPC is used as the policy function approximator to improve the closed-loop performance of target tracking. Simulation experiments and hardware-in-the-loop experiments verified the effectiveness of the proposed method.

     

  • loading
  • [1]
    Dong J, Ota K, Dong M X. UAV-based real-time survivor detection system in post-disaster search and rescue operations[J]. IEEE Journal on Miniaturization for Air and Space Systems, 2021, 2(4): 209-219.
    [2]
    Liu X L, Ansari N. Resource allocation in UAV-assisted M2M communications for disaster rescue[J]. IEEE Wireless Communications Letters, 2019, 8(2): 580-583.
    [3]
    Alharam A, Almansoori E, Elmadeny W, et al. Real time AI-based pipeline inspection using drone for oil and gas industries in Bahrain[C]//Proceedings of the International Conference on Innovation and Intelligence for Informatics, Computing and Technologies. Piscataway: IEEE Press, 2020: 1-5.
    [4]
    Shamsoshoara A, Afghah F, Blasch E, et al. UAV-assisted communication in remote disaster areas using imitation learning[J]. IEEE Open Journal of the Communications Society, 2021, 2: 738-753.
    [5]
    Guo Q, Peng J, Xu W Z, et al. Minimizing the longest tour time among a fleet of UAVs for disaster area surveillance[J]. IEEE Transactions on Mobile Computing, 2022, 21(7): 2451-2465.
    [6]
    Dong W J, Qi X Z, Chen Z X, et al. An indoor path planning and motion planning method based on POMDP[C]//Proceedings of the IEEE International Conference on Robotics and Biomimetics. Piscataway: IEEE Press, 2017: 1564-1570.
    [7]
    Trotti F, Farinelli A, Muradore R. An online path planner based on POMDP for UAVs[C]//Proceedings of the European Control Conference. Piscataway: IEEE Press, 2023: 1-6.
    [8]
    Morere P, Marchant R, Ramos F. Sequential Bayesian optimization as a POMDP for environment monitoring with UAVs[C]//Proceedings of the IEEE International Conference on Robotics and Automation. Piscataway: IEEE Press, 2017: 6381-6388.
    [9]
    Deng D H, Wang C W, Wang W D. Joint air-to-ground scheduling in UAV-aided vehicular communication: a DRL approach with partial observations[J]. IEEE Communications Letters, 2022, 26(7): 1628-1632.
    [10]
    Aniceto Dos Santos M A, Vivaldini K C T. A modeling strategy using Bayesian optimization with POMDP for exploration and informative path planning of UAVs in monitoring of forest[C]//Proceedings of the International Conference on Computational Science and Computational Intelligence. Piscataway: IEEE Press, 2022: 451-456.
    [11]
    Silver D, Veness J. Monte-Carlo planning in large POMDPs[C]//Proceedings of the 24th International Conference on Neural Information Processing Systems - Volume 2. New York: ACM, 2010: 2164-2172.
    [12]
    Ye N, Somani A, Hsu D, et al. DESPOT: online POMDP planning with regularization[J]. Journal of Artificial Intelligence Research, 2017, 58(1): 231-266.
    [13]
    Bai H Y, Cai S J, Ye N, et al. Intention-aware online POMDP planning for autonomous driving in a crowd[C]//Proceedings of the IEEE International Conference on Robotics and Automation. Piscataway: IEEE Press, 2015: 454-460.
    [14]
    Kurniawati H, Yadav V. An online POMDP solver for uncertainty planning in dynamic environment[M]//Robotics Research: The 16th International Symposium ISRR. Berlin: Springer, 2016: 611-629.
    [15]
    Priyadarshini G N, Hsu D, Sun L W. DESPOT-alpha: online POMDP planning with large state and observation spaces[C]//Proceedings of the Robotics: Science and Systems XV. Robotics: Science and Systems Foundation, 2019.
    [16]
    Klimenko D, Song J, Kurniawati H. TAPIR: a software toolkit for approximating and adapting POMDP solutions online[C]//Proceedings of the Australasian Conference on Robotics and Automation. Sydney: Australian Robotics and Automation Association, 2014: 28.
    [17]
    Cunningham A G, Galceran E, Eustice R M, et al. MPDM: multipolicy decision-making in dynamic, uncertain environments for autonomous driving[C]//Proceedings of the IEEE International Conference on Robotics and Automation. Piscataway: IEEE Press, 2015: 1670-1677.
    [18]
    Ding W C, Zhang L, Chen J, et al. EPSILON: an efficient planning system for automated vehicles in highly interactive environments[J]. IEEE Transactions on Robotics, 2022, 38(2): 1118-1138.
    [19]
    Adiprawita W, Ahmad A S, Sembiring J, et al. Reinforcement learning with heuristic to solve POMDP problem in mobile robot path planning[C]//Proceedings of the International Conference on Electrical Engineering and Informatics. Piscataway: IEEE Press, 2011: 1-5.
    [20]
    Lee Y, Cai P P, Hsu D. MAGIC: learning macro-actions for online POMDP planning[C]//Proceedings of the Robotics: Science and Systems XVII. Robotics: Science and Systems Foundation, 2021.
    [21]
    Miller S A, Harris Z A, Chong E K P. A POMDP framework for coordinated guidance of autonomous UAVs for multitarget tracking[J]. EURASIP Journal on Advances in Signal Processing, 2009, 2009(1): 724597.
    [22]
    Ragi S, Chong E K P. UAV path planning in a dynamic environment via partially observable Markov decision process[J]. IEEE Transactions on Aerospace and Electronic Systems, 2013, 49(4): 2397-2412.
    [23]
    Yang Q M, Zhang J D, Shi G Q. Modeling of UAV path planning based on IMM under POMDP framework[J]. Journal of Systems Engineering and Electronics, 2019, 30(3): 545-554.
    [24]
    Eaton C M, Chong E K P, Maciejewski A A. Robust UAV path planning using POMDP with limited FOV sensor[C]//Proceedings of the IEEE Conference on Control Technology and Applications. Piscataway: IEEE Press, 2017: 1530-1535.
    [25]
    Cai W Q, Kordabad A B, Esfahani H N, et al. MPC-based reinforcement learning for a simplified freight mission of autonomous surface vehicles[C]//Proceedings of the 60th IEEE Conference on Decision and Control. New York: ACM, 2021: 2990-2995.
    [26]
    Gros S, Zanon M. Data-driven economic NMPC using reinforcement learning[J]. IEEE Transactions on Automatic Control, 2020, 65(2): 636-648.
    [27]
    Esfahani H N, Kordabad A B, Gros S. Approximate robust NMPC using reinforcement learning[C]//Proceedings of the European Control Conference. Piscataway: IEEE Press, 2021: 132-137.
    [28]
    Lagoudakis M G, Parr R. Least-squares policy iteration[J]. Journal of Machine Learning Research, 2003, 4: 1107-1149.
    [29]
    Airaldi F, De Schutter B, Dabiri A. Reinforcement learning with model predictive control for highway ramp metering[EB/OL]. (2025-02-03)[2026-09-11]. https://doi.org/10.48550/arXiv.2311.08820.
    [30]
    Han Z C, Wu Y W, Li T, et al. An efficient spatial-temporal trajectory planner for autonomous vehicles in unstructured environments[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(2): 1797-1814.
    [31]
    Zhang X J, Liniger A, Borrelli F. Optimization-based collision avoidance[J]. IEEE Transactions on Control Systems Technology, 2021, 29(3): 972-983.
    [32]
    He R X, Zhou J Y, Jiang S, et al. TDR-OBCA: a reliable planner for autonomous driving in free-space environment[C]//Proceedings of the American Control Conference. Piscataway: IEEE Press, 2021: 2927-2934.
    [33]
    Wang Y X, Wang H L, Lun Y B. Shortest path planning of UAV for target tracking and obstacle avoidance in 3D environment[C]// Proceedings of the 39th Chinese Control Conference. Piscataway: IEEE Press, 2020: 3385-3390.
    [34]
    Das M, Ghosh S K. Reducing parameter value uncertainty in discrete Bayesian network learning: a semantic fuzzy Bayesian approach[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2021, 5(3): 361-372.
    [35]
    Youn W, Ko N Y, Gadsden S A, et al. A novel multiple-model adaptive Kalman filter for an unknown measurement loss probability[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 8500511.
    [36]
    Partovibakhsh M, Liu G J. An adaptive unscented Kalman filtering approach for online estimation of model parameters and state-of-charge of lithium-ion batteries for autonomous mobile robots[J]. IEEE Transactions on Control Systems Technology, 2015, 23(1): 357-363.
    [37]
    Dong F, Li X C, You K Y, et al. Standoff tracking using DNN-based MPC with implementation on FPGA[J]. IEEE Transactions on Control Systems Technology, 2023, 31(5): 1998-2010.
    [38]
    Bauersfeld L, Spannagl L, Ducard G J J, et al. MPC flight control for a tilt-rotor VTOL aircraft[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021, 57(4): 2395-2409.
    [39]
    Feng Z W, Cao G Y, Grigoriadis K M, et al. Secure MPC-based path following for UAS in adverse network environment[J]. IEEE Transactions on Industrial Informatics, 2023, 19(11): 11091-11101.
    [40]
    Jacquet M, Kivits M, Das H, et al. Motor-level N-MPC for cooperative active perception with multiple heterogeneous UAVs[J]. IEEE Robotics and Automation Letters, 2022, 7(2): 2063-2070.
    [41]
    Liu Y Y, Van Schijndel J, Longo S, et al. UAV energy extraction with incomplete atmospheric data using MPC[J]. IEEE Transactions on Aerospace and Electronic Systems, 2015, 51(2): 1203-1215.
    [42]
    Falugi P, Mayne D Q. Getting robustness against unstructured uncertainty: a tube-based MPC approach[J]. IEEE Transactions on Automatic Control, 2014, 59(5): 1290-1295.
    [43]
    Xiao K, Tan S C, Wang G H, et al. XTDrone: a customizable multi-rotor UAVs simulation platform[C]//Proceedings of the 4th International Conference on Robotics and Automation Sciences. Piscataway: IEEE Press, 2020: 55-61.
    [44]
    Xiao K, Ma L, Tan S C, et al. Implementation of UAV coordination based on a hierarchical multi-UAV simulation platform[C]//Proceedings of the Advances in Guidance, Navigation and Control. Singapore: Springer Singapore, 2022: 5131-5143.
    [45]
    Zhou B Y, Gao F, Wang L Q, et al. Robust and efficient quadrotor trajectory generation for fast autonomous flight[J]. IEEE Robotics and Automation Letters, 2019, 4(4): 3529-3536.
    [46]
    Yang J, Liu C J, Coombes M, et al. Optimal path following for small fixed-wing UAVs under wind disturbances[J]. IEEE Transactions on Control Systems Technology, 2021, 29(3): 996-1008.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)  / Tables(2)

    Article Metrics

    Article views(37) PDF downloads(39) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return