On the Order Sensitivity of Cascaded Hybrid Anomaly Detection Pipelines: A Shapley-Theoretic Framework with Leakage-Audited Empirical Evaluation

Hybrid anomaly detection pipelines that compose dimensionality-reducing front-ends, densitybaseddetectors, and supervised classifiers are increasingly common, yet the effect of componentorder within such pipelines remains poorly characterized. Prior empirical work has reportedlarge performance swings under permutation of the same components but has neither isolatedthe mechanism nor provided a principled attribution. We introduce a formal frameworkthat recasts a K-stage cascade as a cooperative game whose players are the algorithmicprimitives and whose payoff is a held-out detection score. Within this framework, orderingeffects are quantified by a permutation Shapley value ϕordi for each primitive i, and theresulting train-time distribution shift is bounded via a χ2-divergence between the unfilteredpopulation and the filtered intermediate distribution. We prove a generalization boundfor the terminal classifier in a cascade that scales with the chi-square divergence inducedby upstream filters, formalizing a purity-coverage trade-off. We then conduct a leakageauditedempirical study on 36 tabular benchmarks from ADBench, with hyperparameteroptimization wrapped in nested cross-validation and 30 independent repetitions per pipeline.Across 7,776 permutation-evaluation pairs, we find that order effects are real but modest(median absolute change in PR-AUC of 0.024), that Sparse-Autoencoder-first orderings arestatistically dominated by Isolation-Forest-first orderings on dense tabular data (p < 10−3,Wilcoxon), and that Shapley attribution localizes the marginal value of each primitive in away that classical ablation does not. Several previously reported “hybrid pipeline” results inthe literature are reproducible only under specific leakage configurations; we audit three suchconfigurations and quantify the inflation. Code and a reproducibility appendix are released.