Using a Recurrent Neural Network and Restricted Boltzmann Machines for Malicious Traffic Detection

In the studies of intrusion detection/prevention systems (IDS/IPS) and network security situational awareness, malicious traffic detection has been given significantly more attention to prevent malicious traffic. Meanwhile, with the development of machine learning technology, an increasing number of algorithms and models have been employed for attack detection. Previous studies generally used common and typical machine learning models such as SVM, KNN, or a random forest. However, the bottleneck of these types of approaches is two-fold. The input of the model is constructed using the feature engineering method of artificially designed representation, which requires a substantial amounts expertise. Additionally, most detection methods ignore the temporal information between network packets in one micro-flow. In this paper, we regard malicious traffic detection as a classification task and propose a hybrid model that combines a recurrent neural network (RNN) with restricted Boltzmann machines (RBM) which take byte-level raw data as input without feature engineering. Specifically, distributed embedding is utilized to pre-process network data to make it more suitable for deep neural network models. Subsequently, an RBM model is used to extract the feature vectors of the network packets and an RNN model is used to extract the flow feature vector. Finally, the flow vectors are sent to the Softmax layer to obtain the detection result. Experiments based on the ISCX-2012 and DARPA-1998 published datasets show that our proposed RNN-RBM model has a greater detection accuracy, recall rate, and lower false alarm rate than most traditional machine learning models. This proves the effectiveness of the proposed RNN-RBM model in malicious traffic detection.