Unsupervised Online Anomaly Detection With Parameter Adaptation for KPI Abrupt Changes

IT companies need to monitor various Key Performance Indicators (KPIs) and detect anomalies in real time to ensure the quality and reliability of Internet-based services. However, due to the diversity of KPIs, the ambiguity and scarcity of anomalies and the lack of labels, anomaly detection for various KPIs has been a great challenge. Existing KPI anomaly detection methods have not explored the properties of anomalies in KPIs in detail to our best knowledge. Therefore, we explore anomalies in KPIs and recognize a common and important form of anomalies named abrupt changes, which often indicate potential failures in the relevant services. For abrupt changes in various KPIs, we propose DDCOL, an unsupervised online anomaly detection algorithm with parameter adaptation from the perspective of anomalies for the first time. We propose three techniques: high order D ifference extraction and combination, D ensity-based C lustering with parameter adaptation and OnL ine detection with subsampling (DDCOL). Compared with traditional statistical methods and unsupervised learning methods, extensive experimental results and analysis on a large number of public KPIs show the competitive performance of DDCOL and the significance of abrupt changes. Furthermore, we provide an interpretation for the promising results, which shows that DDCOL can be robust to KPI expected concept drifts, and obtain a good feature distribution of normal data in KPIs.