Fast entropy based alert detection in super computer logs

The task of alert detection in event logs is very important in preventing or recovering from downtime events. The ability to do this automatically and accurately provides significant savings in the time and cost of downtime events. The Nodeinfo algorithm, which is currently in production use at Sandia National Laboratories, is an entropy based algorithm for alert detection in event logs. Automatic alert detection needs to be fast for it to be practical in a production environment. In this work we show that with Message Type Indexing (MTI) the computational effort required for alert detection can be reduced by up to 99%. This can be achieved without a drop in detection performance. Our proposed method has special significance because it provides a framework for alert detection which requires little or no human input, due to message type extraction required for MTI being carried out automatically using the Iterative Partitioning Log Mining (IPLoM) algorithm.