Cross-Dataset Evaluation of Intrusion Detection Models for IoT Networks

As IoT networks become more complex, so too does their vulnerability to various cyber threats and attacks, making the need for strong IDS an absolute must. This paper investigates IDS machine learning models based on both a single data set as well as cross-boundary datasets. Supervised, unsupervised, and hybrid methods (Random Forest, Isolation Forest, Autoencoder, etc.), as well as their combinations, were evaluated using UNSW-NB15 and CICIDS2017 datasets. Where most studies evaluate models using training and testing sets within the same data set, this study primarily evaluated their ability to generalize to a different dataset. Based on experimental results, supervised models performed well for training and testing within the same data set; however, their ability to generalize was drastically reduced when training and testing were performed across datasets (~90% F1 score for r. Forest vs. 52% F1 score for r. Forest when trained/tested on same dataset). Conversely, while unsupervised models lack accuracy when training/testing on controlled data sets, they do show relatively consistent performance when comparing their performance across different data distributions due to dataset distribution biases. Based upon feature distribution comparisons among data distributions, it appears that differences in dataset distributions have the greatest effect on the generalization of current IDS models. This study concluded that there are limitations to the current state-of-the-art methods of IDS and that in any design of an IDS for a real-world IOT scenario; dataset distribution shifts must be accounted for when providing for reliable INS.