-
摘要:
针对传统多智能体深度确定性策略梯度(MADDPG)算法在多无人机(UAV)航迹规划中存在的探索效率低、价值估计偏差大及训练稳定性不足等问题，提出一种改进的MADDPG算法。该算法引入双延迟深度确定性策略梯度(TD3)的核心机制，包括双价值网络、延迟策略更新及目标策略平滑，并结合探索噪声的指数衰减策略，从而在保持策略多样性的同时提升收敛稳定性。在此基础上，分别设计适用于多无人机协同航迹规划的状态空间与动作空间，构建密集型奖励函数，并实现高效稳定的路径规划策略。通过在三维静态环境下的仿真实验，对比传统MADDPG算法与所提改进MADDPG算法的性能差异。结果表明：所提改进MADDPG算法能够在多种起止位置及障碍分布条件下实现快速收敛与稳定规划，相较于传统MADDPG算法在路径效率、任务完成率及协同控制能力等方面均具有显著优势，验证了其在复杂空域多无人机协同航迹规划任务中的有效性与鲁棒性。
Abstract:To address the issues of low exploration efficiency, value estimation bias, and insufficient training stability in the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm for multi- nmanned aerial vehicle（UAV）trajectory planning, this paper proposes an improved MADDPG algorithm. To preserve policy diversity while improving convergence stability, the suggested approach combines an exponentially decaying exploration noise strategy with the fundamental mechanisms of the twin delayed deep deterministic policy gradient (TD3), such as a dual-critic network, delayed policy updates, and target policy smoothing. Furthermore, tailored state and action spaces are designed for multi-UAV cooperative trajectory planning, along with a dense reward function to ensure efficient and stable path generation. A three-dimensional static simulation environment is constructed to train and comparatively evaluate the proposed improved MADDPG method against the traditional MADDPG. Experimental results demonstrate that the proposed improved MADDPG algorithm achieves rapid convergence and stable planning performance under various starting/ending positions and obstacle distributions. It validates its efficacy and robustness for cooperative multi-UAV trajectory planning in complicated airspace scenarios by achieving notable gains in path efficiency, task completion rate, and cooperative control capability when compared to the old technique.
-
图 5 训练阶段平均步数对比
Figure 5. Average episode steps per round comparison during training phase
图 6 不同训练轮次下无人机航迹对比
Figure 6. Trajectory comparison of UAVs in different training episodes
图 7 动态障碍场景下3种算法无人机航迹对比
Figure 7. Trajectory comparison of UAVs under dynamic obstacle scenarios for three algorithms
表 1 MADDPG超参数设置
Table 1. Key Hyperparameters for MADDPG-Based Simulation
参数 设置值 作用说明 折扣因子$ \gamma $ 0.99 未来奖励的衰减系数 批量大小（Batch_size） 256 经验回放采样批次大小 学习率（Actor/Critic） 0.001/0.002 Adam优化器参数 经验回放池容量 100 000 存储状态-动作-奖励样本 最大训练回合数 5 000 训练终止条件 单回合最大步数 100 最大步数限制防止无限循环 优化器 Adam 参数更新算法 高斯探索噪声 $ N(0,{0.6}^{2}) $ 控制动作输出的随机性 
下载: 导出CSV
-
[1] 陈雯琦. 多无人机协同目标追踪的路径规划研究[D]. 西安: 西安工业大学, 2024: 1-69.Chen W Q. Research on path planning for collaborative target tracking of multiple unmanned aerial vehicles[D]. Xi’an: Xi’an Technological University, 2024: 1-69(in Chinese). [2] 李升虎. 基于改进深度强化学习算法的多无人机路径规划研究[D]. 曲阜: 曲阜师范大学, 2023: 1-76.Li S H. Research on multi-UAV path planning based on improved deep reinforcement learning algorithm[D]. Qufu: Qufu Normal University, 2023: 1-76(in Chinese). [3] Cheng X, Shi H G, Jin Z Q, et al. Energy efficiency aware collaborative multi-UAV deployment for intelligent traffic surveillance[C]//Proceedings of the IEEE Latin-American Conference on Communications. Piscataway: IEEE Press, 2023: 1-6. [4] Huang S Y, Zhang H B, Huang Z Y. Multi-UAV collision avoidance using multi-agent reinforcement learning with counterfactual credit assignment[EB/OL]. (2025-07-14)[2025-09-02]. https://arxiv.org/abs/2204.08594v1. [5] Wang Z, Xiang X B. Improved astar algorithm for path planning of marine robot[C]//Proceedings of the 37th Chinese Control Conference. Piscataway: IEEE Press, 2018: 5410-5414. [6] Libby J, Stentz A. Multiclass terrain classification using sound and vibration from mobile robot terrain interaction[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE Press, 2021: 2305-2312. [7] Bolourian N, Hammad A. LiDAR-equipped UAV path planning considering potential locations of defects for bridge inspection[J]. Automation in Construction, 2020, 117: 103250. [8] 滕菲, 王迎春, 姚永辉, 等. 基于深度强化学习的无人机动态避障规划[J/OL]. 北京航空航天大学学报, (2025-05-23)[2025-09-02]. https://doi.org/10.13700/j.bh.1001-5965.2025.0084.Teng F, Wang Y C, Yao Y H, et al. Dynamic obstacle avoidance planning for UAV based on deep reinforcement learning[J/OL]. Journal of Beijing University of Aeronautics and Astronautics, (2025-05-23)[2025-09-02]. https://doi.org/10.13700/j.bh.1001-5965.2025.0084(in Chinese). [9] Yu C, Velu A, Vinitsky E, et al. The surprising effectiveness of PPO in cooperative multi-agent games[C]//Proceedings of the 36th International Conference on Neural Information Processing Systems. New York: ACM, 2022: 24611-24624. [10] Watkins C J C H. Learning from delayed rewards[D]. Cambridge: King's College, 1989. [11] Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[EB/OL]. (2013-12-19)[2025-09-02]. https://arxiv.org/abs/1312.5602. [12] Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[EB/OL]. (2019-07-05)[2025-09-02]. https://arxiv.org/abs/1509.02971. [13] Lowe R, Wu Y, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 6382-6393. [14] Rashid T, Samvelyan M, De Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(1): 7234-7284. [15] Fujimoto S, Van Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//Proceedings of the International Conference on Machine Learning. Stockholm: PMLR, 2018: 1587-1596. -
下载:
点击查看大图
计量
- 文章访问数: 221
- HTML全文浏览量: 78
- PDF下载量: 7
- 被引次数: 0
