Hybrid PER–TD3–HER reinforcement learning for joint UAV trajectory, IRS phase, and resource allocation in energy-harvesting cell-free IoT networks

This paper proposes a customized hybrid deep reinforcement learning framework, termed PER–TD3–HER, for joint optimization of three-dimensional UAV trajectory design, intelligent reflecting surface (IRS) phase configuration, transmit power allocation, and time-switching energy harvesting in UAV-assisted, IRS-enabled energy-harvesting cell-free IoT networks. The proposed framework integrates the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay (PER) and Hindsight Experience Replay (HER) mechanisms to improve learning efficiency, exploration capability, and robustness in high-dimensional continuous control environments. Unlike conventional approaches that optimize network components separately, the proposed framework provides a unified learning-based solution for the coupled optimization of UAV mobility, IRS-assisted propagation control, resource allocation, and energy sustainability under dynamic network conditions and imperfect channel state information (CSI). In the proposed framework, PER enhances sample efficiency by prioritizing informative experiences associated with critical learning transitions, QoS requirements, energy constraints, and challenging channel conditions. Meanwhile, HER improves goal-oriented learning by transforming unsuccessful trajectories into valuable training experiences through goal relabeling. Furthermore, the continuous action-space formulation enables adaptive and simultaneous control of UAV trajectory, IRS phase shifts, transmit power allocation, and energy-harvesting ratios, overcoming the limitations of discrete decision-making approaches. Comprehensive simulations under realistic channel conditions and imperfect CSI demonstrate that the proposed PER–TD3–HER framework achieves improved convergence behavior, throughput–energy trade-off, and QoS satisfaction compared with TD3-based, DDPG-based, DQN-based, and SAC benchmark methods. The results verify the effectiveness, robustness, and scalability of the proposed framework for sustainable UAV-assisted IRS-enabled cell-free IoT networks and its potential applicability to future intelligent wireless communication systems.