Reinforcement Learning Method for Collaborative Multi-Objective Path Planning of IIoT Drones and Unmanned Vehicles in Random Demand and Dynamic Delivery Environments
DOI:
https://doi.org/10.4108/eetsis.13700Keywords:
Industrial Internet of Things, Drone-Unmanned Vehicle Collaboration, Stochastic Demand, Multi-Objective Path Planning, MADDPG, Guidance FactorAbstract
INTRODUCTION: Dynamic Industrial Internet of Things (IIoT) delivery is characterized by stochastic order arrivals, heterogeneous UAV–UGV capabilities, and real-time demand updates, making conventional static routing methods insufficient for collaborative delivery planning. Existing UAV trajectory-planning and truck–drone routing studies usually focus on either single-platform trajectory control or offline vehicle routing, and they rarely integrate stochastic IIoT demand updating, UAV–UGV rendezvous coordination, operational constraints, and multi-objective decision-making into a unified reinforcement-learning framework.
OBJECTIVES: To address this limitation, this study proposes a guidance-factor-enhanced MADDPG framework for collaborative multi-objective path planning of UAVs and UGVs in random-demand and dynamic delivery environments.
METHODS: Order arrivals are modeled by a Poisson process, while order location, payload, priority, and time-window attributes are generated through multivariate random distributions and updated by IIoT terminals every 5 s. Payload, endurance, speed, rendezvous, and task-sequencing constraints are incorporated into a normalized multi-objective function considering delivery time, delivery cost, demand satisfaction, and UAV endurance loss. A distance–demand–priority guidance factor and a reconstructed reward function are further introduced to alleviate sparse-reward effects and guide agents toward high-value demand regions.
RESULTS: Simulation and ablation results show that the reconstructed reward is essential for stable learning, while the guidance factor significantly accelerates convergence. Under 40 IIoT nodes, the proposed method converges after approximately 2,000 episodes, whereas the model without guidance information requires about 5,000 episodes. Comparative experiments further show that the proposed method improves system throughput and collaborative delivery efficiency while maintaining feasible UAV trajectories.
CONCLUSION: The proposed framework provides an adaptive reinforcement-learning solution for UAV–UGV cooperative logistics in dynamic IIoT environments.
References
[1] SISINNI E, SAIFULLAH A, HAN S, et al. Industrial Internet of Things: Challenges, opportunities, and directions[J]. IEEE Transactions on Industrial Informatics, 2018, 14(11): 4724–4734.
[2] XU X, LU Y, VOGEL-HEUSER B, et al. Industry 4.0 and Industry 5.0—Inception, conception and perception[J]. Journal of Manufacturing Systems, 2021, 61: 530–535.
[3] MADDIKUNTA P K R, PHAM Q V, PRABADEVI B, et al. Industry 5.0: A survey on enabling technologies and potential applications[J]. Journal of Industrial Information Integration, 2022, 26: 100257.
[4] ZHAO G, WANG Y, MU T, et al. Reinforcement-learning-assisted multi-UAV task allocation and path planning for IIoT[J]. IEEE Internet of Things Journal, 2024, 11(16): 26766–26777.
[5] HAN J, LIU Y, LI Y. Vehicle routing problem with drones considering time windows and dynamic demand[J]. Applied Sciences, 2023, 13(24): 13086.
[6] CUI H, LI K, JIA S, et al. Dynamic collaborative truck-drone delivery with en-route synchronization and random requests[J]. Transportation Research Part E: Logistics and Transportation Review, 2024, 192: 103802.
[7] WANG F, LI H, XIONG H. Truck-drone routing problem with stochastic demand[J]. European Journal of Operational Research, 2025, 322(3): 854–869.
[8] NING Z, YANG Y, WANG X, et al. Multi-agent deep reinforcement learning based UAV trajectory optimization for differentiated services[J]. IEEE Transactions on Mobile Computing, 2024, 23(5): 5818–5834.
[9] FAN C, XU H, WANG Q. Multi-agent deep reinforcement learning for trajectory planning in UAVs-assisted mobile edge computing with heterogeneous requirements[J]. Computer Networks, 2024, 248: 110469.
[10] JU T, LI L, LIU S, et al. A multi-UAV assisted task offloading and path optimization for mobile edge computing via multi-agent deep reinforcement learning[J]. Journal of Network and Computer Applications, 2024, 229: 103919.
[11] KANG H, CHANG X, MIŠIĆ J, et al. Cooperative UAV resource allocation and task offloading in hierarchical aerial computing systems: A MAPPO-based approach[J]. IEEE Internet of Things Journal, 2023, 10(12): 10497–10509.
[12] SUN H, ZHOU Y, ZHANG H, et al. Joint optimization of caching, computing and trajectory planning in aerial mobile edge computing networks: An MADDPG approach[J]. IEEE Internet of Things Journal, 2024, 11(24): 40996–41007.
[13] HWANG S, LEE H, KIM M, et al. Multi-agent deep reinforcement learning for decentralized multi-UAV mobile edge computing networks[J]. IEEE Internet of Things Journal, 2025, 12(10): 14484–14497.
[14] XU S, LIU Q, GONG C, et al. Energy-efficient multi-agent deep reinforcement learning task offloading and resource allocation for UAV edge computing[J]. Sensors, 2025, 25(11): 3403.
[15] Roijers, Diederik M., et al. "A survey of multi-objective sequential decision-making." Journal of Artificial Intelligence Research 48. 2013: 67-113.
[16] Van Moffaert, Kristof, and Ann Nowé. "Multi-objective reinforcement learning using sets of pareto dominating policies." The Journal of Machine Learning Research 15.1. 2014: 3483-3512.
[17] Pirotta, Matteo, Simone Parisi, and Marcello Restelli. "Multi-objective reinforcement learning with continuous pareto frontier approximation." Proceedings of the AAAI conference on artificial intelligence. Vol. 29. No. 1. 2015.
[18] Zhang, Kaiqing, et al. "Robust multi-agent reinforcement learning with model uncertainty." Advances in neural information processing systems 33. 2020: 10571-10583.
[19] LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Advances in Neural Information Processing Systems. 2017: 6379–6390.
[20] LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[EB/OL]. arXiv:1509.02971, 2015.
[21] NG A Y, HARADA D, RUSSELL S. Policy invariance under reward transformations: Theory and application to reward shaping[C]//Proceedings of the 16th International Conference on Machine Learning. 1999: 278–287.
[22] Saini, Hemant Kumar. "Connecting the 6G Autonomous Worlds with Real Time Edge Intelligence (Autonomous Vehicle)." Edge Intelligence for 6G‐Enabled Industrial Internet of Things (2026): 163-180.
[23] Rakhra, Manik, Deepak Prashar, and Sudan Jha. "A Graph-based AI Generated Visual Manipulation Detector for UAV-based Industrial Systems." Cognitive Security for Industrial IoT. CRC Press, 2026. 110-124.
[24] Rashmi, K. B., et al. "Intrusion Detection for Unmanned Aerial Vehicles (UAVs) via Lightweight Federated Continuous Learning." 2026 5th International Conference on Sentiment Analysis and Deep Learning (ICSADL). IEEE, 20
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Li Qiao, Changkai Xu, Bin Ni
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open access article distributed under the terms of the CC BY-NC-SA 4.0, which permits copying, redistributing, remixing, transformation, and building upon the material in any medium so long as the original work is properly cited.
