Hierarchical Deep Reinforcement Learning framework for multi-year asset management under budget constraints

Budget planning and maintenance optimisation are crucial for infrastructure asset management, ensuring cost-effectiveness and sustainability. However, the complexity arising from combinatorial state-action spaces, diverse asset deterioration, stringent budget constraints and environmental uncertainty significantly limits existing methods’ capability of dealing with large-scale systems, or simply, scalability. This paper proposes a Hierarchical Deep Reinforcement Learning (HDRL) methodology specifically tailored to multi-year infrastructure planning. The proposed approach decomposes the problem into two hierarchical levels: a high-level Budget Planner allocating annual budgets within explicit feasibility bounds and a low-level Intervention Planner prioritising assets within the allocated annual budget. By structurally separating macro-budget decisions from asset-level prioritisation and integrating linear programming projection within a hierarchical Soft Actor-Critic framework, the method efficiently addresses exponential growth in the action space and ensures rigorous budget compliance. A case study evaluating sewer networks of varying sizes (10, 15 and 20 sewersheds) illustrates the effectiveness of the proposed approach. Compared to conventional Deep Q-Learning and enhanced genetic algorithms, the proposed methodology converges more rapidly, scales effectively and consistently delivers near-optimal solutions even as network size grows.