Making APT Detection Research Reproducible: A Standardized Evaluation Protocol and Unified Scoring System for Provenance-Based Security Tools

Advanced Persistent Threat (APT) detection systems based on provenance graphs have achieved strong performance in research environments, but lack reproducibility due to inconsistent experimental configurations and fragmented evaluation practices. This work introduces ProvEval, a standardized reproducibility protocol and unified evaluation framework for provenance-based intrusion detection systems. It defines structured experimental requirements across hardware, software, dataset, hyperparameters, and evaluation settings to enable independent replication of results. We further propose IOUS (Integrated Operational Utility Score), a multi-dimensional metric that combines detection accuracy, investigation quality, operational feasibility, and adversarial robustness into a single interpretable score for cross-system comparison. The framework is implemented and demonstrated using state-of-the-art systems including MAGIC and R-CAID, highlighting significant gaps in reproducibility and evaluation standardization in current research.