Adaptive Multi‐Axis Partitioning and Data Pruning for Parallel Outlier Detection in Large‐Scale High‐Dimensional Data

With the explosive growth of large‐scale, high‐dimensional data in fields such as finance, IoT, and bioinformatics, parallel outlier detection has become essential for efficiently identifying outliers while enabling scalable processing. However, most existing approaches suffer from high interdependency between computational modules, which leads to inefficient resource utilization and escalated I/O overhead. Additionally, their load balancing strategies fail to account for the distribution characteristics of data attributes, hindering the detection of outliers in sparsely distributed attribute subsets. In this paper, we propose a novel parallel outlier detection framework, named HODAP‐DP, which integrates adaptive multi‐axis recursive partitioning and local I/O performance optimization strategies. First, an adaptive multi‐axis recursive partitioning strategy is designed by exploiting the statistical properties of data attribute distributions, so that load balancing across partitions is ensured and outliers within sparse subsets can be effectively detected. Second, we introduce a local I/O performance optimization strategy leveraging data density locality and data pruning in memory pipelining to reduce cross‐node communication and I/O overhead caused by interdependent computation, thereby enhancing the utilization of computational and memory resources. Third, a novel parallel outlier detection algorithm, which employs Chebyshev Coulomb resultant force as the similarity measures, is proposed by integrating recursive multi‐axis partitioning and local I/O performance optimization. In the end, experimental results on UCI, TIGER, and TB‐scale datasets validate the algorithm's superior performance, scalability, and extensibility.