Toward deployment-oriented benchmarking of intrusion detection systems: integrating threshold dynamics, distribution shift, and interpretability into comparative model evaluation

Intrusion detection systems (IDSs) constitute a foundational layer of contemporary cyber-defense infrastructures; however, the methodological rigor of IDS benchmarking remains constrained by evaluation protocols that frequently rely on isolated metrics, static experimental conditions, or narrowly defined operational assumptions. Such practices often obscure critical discrepancies between statistical discrimination capability and deployment-relevant decision performance. In this study, we introduce SH4 (Scenario-aware, Heterogeneity-focused, Hybrid, and Holistic benchmarking framework), a comprehensive benchmarking framework designed to enable scenario-aware, multi-dimensional, and operationally grounded evaluation of heterogeneous IDS architectures, including supervised tabular learners, sequence-oriented neural models, and proxy deep detection mechanisms. Unlike conventional benchmarking pipelines, SH4 integrates threshold-dependent and threshold-independent assessment, repeated stratified validation, drift-aware online evaluation, adaptive threshold analysis, controlled ablation experiments, and interpretability-oriented diagnostics within a unified benchmarking pipeline. This design permits systematic examination of model robustness under varying operating conditions while exposing latent trade-offs that are typically concealed by single-score reporting practices. Empirical evaluation showed that multilayer perceptron and reconstruction-based proxy detection achieved the strongest overall performance under the primary benchmark configuration, while the baseline, sequence-based models, and CNNLSTMProxy also remained competitive across multiple complementary metrics. By contrast, additional scenario-based analyses suggested that several intermediate and comparatively weaker detectors did not always preserve the same relative standing across ranking-oriented and threshold-dependent evaluation criteria. A benchmark-centered lower-separability sensitivity analysis further suggested that the principal comparative ordering remained broadly stable under progressively reduced separability. Furthermore, sequence-based architectures displayed pronounced sensitivity to sequence construction strategies and temporal representation choices, suggesting that reported advantages of sequential modeling may be strongly contingent upon preprocessing assumptions and evaluation design. Our analyses additionally demonstrate that threshold calibration and distributional drift can materially alter the practical utility and reliability of several detectors, particularly baseline and proxy-based models. Collectively, these findings challenge the continued reliance on single headline metrics as indicators of IDS effectiveness. Within the scope of the evaluated synthetic and surrogate benchmarking conditions, the results suggest that deployment-oriented IDS assessment may benefit from jointly considering discrimination capacity, threshold behavior, robustness under distributional shift, scenario sensitivity, and interpretability to support more transparent and informed comparative evaluation.