Subcluster: towards scalable and efficient KPIs clustering for cybersecurity operation via representative subsequence extraction

Modern cybersecurity infrastructure management is facing the challenge of an explosion in the number of machines and Key Performance Indicators(KPIs). As a fundamental data-mining technique, clustering can not only group data with similar characteristics but also provide informative insights that are critical for cybersecurity operations. In particular, the clustering of large-scale KPIs can significantly assist cybersecurity engineers in detecting system failures, reducing redundant alarms, and decreasing troubleshooting overhead-contributing to more effective threat detection and alert management. However, KPIs are reasonably long (that is, high dimensionality) because of the continuous collection of the monitoring system. For KPIs’ high dimensionality, existing KPIs clustering methods always have a trade-off between clustering quality and computational complexity. In this paper, we propose a novel method, SubCluster, which is a fast, accurate, and scalable approach for clustering large-scale KPIs with high quality. SubCluster aims to extract representative subsequences from numerous KPIs. In contrast to existing subsequence extraction-based clustering methods that compress the candidate set, SubCluster first conducts coarse-grained clustering with the nearest neighbor-based graph to obtain near-accurate groups from raw KPIs. SubCluster then attempts to extract representative subsequences for fine-grained clustering. Extensive experimental results on public time-series and large-scale KPIs datasets from the real world demonstrate the effectiveness and efficiency of SubCluster.