Multi-Agent Reinforcement Learning for Anomaly Detection with Hierarchical Adaptive Sampling Strategy

The proliferation of sensor technologies has enabled real-time monitoring across diverse domains. In many practical systems, sensors are organized into geographically or functionally distinct regions, and continuous monitoring is constrained by limited sensing budgets. Such systems exhibit a hierarchical and resource-constrained data layout, where resources must be intelligently allocated across regions to capture the most informative data streams. We refer to this as the System-Region-Stream (SRS) structure. To support sequential anomaly detection in SRS systems under resource constraints, we formulate the monitoring task as a leader-follower decision-making problem, where a central planner allocates limited resources across regions, and regional detectors autonomously sample data streams for local monitoring. To optimize this hierarchical process, we propose a two-stage trust region learning framework that stabilizes policy updates and establishes monotonic-improvement properties for ideal trust-region updates of both planner and detectors. Building on this scheme, we develop HIPPO, a practical multi-agent deep reinforcement learning (DRL) algorithm that scales to large SRS systems by integrating neural function approximation with constrained action parameterizations for integer-valued resource allocation and budget-aware local sampling. Extensive simulation studies and real-world case studies demonstrate that our proposed method outperforms existing baselines significantly.