A Comparison of Machine Learning Approaches to Detect Botnet Traffic

During the past decade, botnets have become one of the most significant threats in the field of network security. A botnet attack typically works by infecting a device with malware and then recruiting it into a network of infected devices controlled by an attacker, which may lead to severe economic and social consequences. As a result, a considerable amount of research has been conducted to detect and prevent such attacks. In this paper, we create a foundation for an anomaly-based intrusion detection system to improve network security and to help reduce human involvement and error. The network traffic we used is captured in the form of connection logs, which are gen- erated by a popular network monitoring framework called Bro. We use our proposed framework to compare the performances of multiple supervised learning approaches, including Logistic Regression (LR), Naive Bayes (NB), Support Vector Machine (SVM), Random Forest RF and Neural Networks (NN) at anomaly detection. We evaluated these models using F1 Score and Area Under Curve (AUC). Our models are trained on malicious network traffic samples from pervasive botnets like Zeus, Miuref and Conficker, as well as benign traffic samples. Using traditional cross validation, we illustrate that Random Forest has the best performance for anomaly detection. To test each algorithm's ability to generalize to unseen bot types, we implemented a custom-designed Leave-One-Out Cross Validation (LOBO-CV). In this procedure, each algorithm is trained on all but one bot family and evaluated on their ability to detect the unknown botnet traffic. We then improved overall detection performance by ensembling our two best models. Our results demonstrate we can detect both previously seen bot families and unseen botnet families with a high degree of confidence.