LogFlash: Real-time Streaming Anomaly Detection and Diagnosis from System Logs for Large-scale Software Systems

Today, software systems are getting increasingly large and complex and a short failure time may cause huge loss. Therefore, it is important to detect and diagnose anomalies accurately and timely. System logs are a straightforward and important source of information for anomaly detection and diagnosis. However, existing log-based approaches have three key limitations. First, they are not designed for processing real-time log streams. Second, they require restrictions on training log data. Third, they lack the adaptiveness to system update. To break through these limitations, we propose LogFlash, a real-time streaming anomaly detection and diagnosis approach that enables both training and detection in a real-time streaming processing manner. By assigning a dynamic pairwise transition rate to each template pair and model the transition possibility as typical power-law distribution, our approach achieves real-time model construction and updates. Experiment results show that it reduces over 5 times of training and detection time compared with the state-of-art works while maintaining the capability of accurate anomaly diagnosis.