| Citation: | Zhou Y M，Han T，Jiang Y F，et al. A multi-aircraft cooperative air combat decision-making algorithm based on an adaptive weighted priority virtual self-play theory approach[J]. Journal of Beijing University of Aeronautics and Astronautics，2026，52（10）：3580-3591 （in Chinese）
|
This study suggests an approach that combines prioritized fictitious self-play (PFSP) with an adaptive winning-game weighting mechanism to address the delayed convergence and policy oscillation in self-play training for multi-aircraft cooperative beyond-visual-range (BVR) air combat. Firstly, a 4v4 BVR air combat simulation environment is established under a multi-agent reinforcement learning framework. Second, in order to break the policy cycle, the Elo method is developed to conduct non-uniform sampling of past opponents and dynamically assess policy capabilities. Meanwhile, a win-rate-based loss function weighting mechanism is designed to guide gradient updates when early high-value samples are scarce. According to experiments, the suggested approach greatly lessens later-stage oscillation and converges to the Nash equilibrium by the 200th episode, increasing convergence efficiency by 55% over conventional FSP. The weighting mechanism’s accelerating impact is further supported by ablation investigations. Lastly, agents educated using PFSP have strong air combat decision-making ability, winning 70.3% of the time against skilled opponents.
| [1] |
樊会涛, 闫俊. 空战体系的演变及发展趋势[J]. 航空学报, 2022, 43(10): 527397.
Fan H T, Yan J. Evolution and development trend of air combat system[J]. Acta Aeronautica et Astronautica Sinica, 2022, 43(10): 527397(in Chinese).
|
| [2] |
孙智孝, 杨晟琦, 朴海音, 等. 未来智能空战发展综述[J]. 航空学报, 2021, 42(8): 525799.
Sun Z X, Yang S Q, Piao H Y, et al. A survey of air combat artificial intelligence[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(8): 525799(in Chinese).
|
| [3] |
罗越群, 丁达理, 谭目来, 等. 无人作战飞机自主机动决策方法综述[J]. 航空学报, 2025, 46(7): 130884.
Luo Y Q, Ding D L, Tan M L, et al. A review of autonomous maneuver decision methods for unmanned combat aerial vehicle[J]. Acta Aeronautica et Astronautica Sinica, 2025, 46(7): 130884(in Chinese).
|
| [4] |
李波, 黄晶益, 万开方, 等. 基于深度强化学习的无人机系统应用研究综述[J]. 战术导弹技术, 2023(1): 58-68.
Li B, Huang J Y, Wan K F, et al. A review of research on the application of UAV system based on deep reinforcement learning[J]. Tactical Missile Technology, 2023(1): 58-68(in Chinese).
|
| [5] |
方思雨, 张振华, 牛经龙, 等. 多无人机空战机动决策中的深度强化学习: 实践中的关键技术综述与未来展望[J]. 航空兵器, 2025(2): 19-33.
Fang S Y, Zhang Z H, Niu J L, et al. Deep reinforcement learning in multi-UAV air combat maneuver decision-making: a review of key techniques in practice and future prospects[J]. Aero Weaponry, 2025(2): 19-33(in Chinese).
|
| [6] |
Li W T, Fang F, Peng D L, et al. An intelligent maneuver decision-making approach for air combat based on deep reinforcement learning and transformer networks[J]. Entropy, 2024, 26(12): 1036.
|
| [7] |
Yang J, Wang L P, Han J L, et al. An air combat maneuver decision-making approach using coupled reward in deep reinforcement learning[J]. Complex & Intelligent Systems, 2025, 11(8): 364.
|
| [8] |
Wang H M, Zhou Z F, Jiang J Z, et al. Autonomous air combat maneuver decision-making based on PPO-BWDA[J]. IEEE Access, 2024, 12: 119116-119132.
|
| [9] |
Mei J R, Li G, Huang H S. Deep reinforcement-learning-based air-combat-maneuver generation framework[J]. Mathematics, 2024, 12(19): 3020.
|
| [10] |
Zhang X, Dong W H, Zhang P, et al. Research on the reward design method for deep reinforcement learning in WVR air combat[J]. IEEE Access, 2024, 12: 182693-182707.
|
| [11] |
Zhu J Y, Kuang M C, Zhou W Q, et al. Mastering air combat game with deep reinforcement learning[J]. Defence Technology, 2024, 34: 295-312.
|
| [12] |
Gao X Z, Zhang Y, Wang B L, et al. The optimal strategies of maneuver decision in air combat of UCAV based on the improved TD3 algorithm[J]. Drones, 2024, 8(9): 501.
|
| [13] |
Shi G Q, Cao Y, Wang D H, et al. Towards intelligent unmanned adversarial games: a reinforcement learning framework with the PHP-ROW method[J]. Drones, 2025, 9(5): 331.
|
| [14] |
单圣哲, 张伟伟. 基于自博弈深度强化学习的空战智能决策方法[J]. 航空学报, 2024, 45(4): 328723.
Shan S Z, Zhang W W. Air combat intelligent decision-making method based on self-play and deep reinforcement learning[J]. Acta Aeronautica et Astronautica Sinica, 2024, 45(4): 328723(in Chinese).
|
| [15] |
Li B, Huang J Y, Bai S X, et al. Autonomous air combat decision-making of UAV based on parallel self-play reinforcement learning[J]. CAAI Transactions on Intelligence Technology, 2023, 8(1): 64-81.
|
| [16] |
Kong W R, Zhou D Y, Zhou Y, et al. Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat[J]. Journal of Computational Design and Engineering, 2023, 10(2): 830-859.
|
| [17] |
Tasbas A S, Sahin S O, Üre N K. Reinforcement learning based self-play and state stacking techniques for noisy air combat environment: AIAA 2023-1077[R]. Reston: AIAA, 2023.
|
| [18] |
Yang M C, Shan S Z, Zhang W W. Decision-making and confrontation in close-range air combat based on reinforcement learning[J]. Chinese Journal of Aeronautics, 2025, 38(9): 103526.
|
Figures(5) / Tables(5)
Copyright © Journal of Beijing University of Aeronautics and Astronautics
Address: Editorial Department of Journal of Beijing University of Aeronautics and Astronautics, 37 Xueyuan Road, Haidian District, Beijing Post Code: 100191 Email: jbuaa@buaa.edu.cn
Supported by:
Beijing Renhe Information Technology Co., Ltd.
DownLoad:
