GULL: Graph-Guided Hierarchical Reinforcement Learning for Intelligent Aerial Ferry Routing in Digital Twins

Operating Unmanned Aerial Vehicles (UAVs) in volatile, infrastructure-denied environments poses a fundamentally multi-scale challenge: a Ferry UAV must make long-horizon data-ferrying decisions while reacting in real time to terrain constraints, topology uncertainty, and physical disturbances. This paper presents Graph-guided UAV Layered Learning (GULL), a single-agent hierarchical reinforcement learning framework for intelligent aerial data ferrying in dynamic multi-UAV networks. Specifically, GULL combines graph-guided reasoning over dynamic network nodes with a layered decision architecture that decouples long-horizon strategic routing from short-horizon tactical motion control. To support such multi-scale decision making, GULL integrates self-attention-based aggregation of dynamic node features with a dual-resolution spatial representation encoded by convolutional modules, where Demand, Entropy, and self-supervised Cost maps jointly capture service urgency, topological uncertainty, and terrain traversability. Built upon this multi-modal representation, a hierarchical policy is further equipped with an Adaptive Temporal Abstraction (ATA) mechanism that dynamically adjusts high-level decision horizons to balance delivery objectives against kinematic and energy constraints. To improve policy generalization, GULL is trained in a Digital Twin augmented by a Large Language Model (LLM) that automatically generates diverse, high-fidelity terrains. Extensive experiments show that GULL improves the priority-weighted delivery ratio by up to 47% and reduces average end-to-end latency by 50% compared with representative baselines, while exhibiting strong scalability and robustness under severe traffic congestion, packet loss, wind disturbance, and complex terrain conditions.