A hybrid CNN-LSTM framework for real-time web anomaly detection using incremental feature reduction

The exponential growth of web based services have required more sophisticated security architectures capable of detecting high velocity, high dimensional, multi-stage cyber attacks. Traditional rule based defenses and static machine learning models suffer from a “Real Time Paradox”, where high accuracy detection comes at the expense of insufferably high computational latency. The conceptual architecture involves an application of Incremental Principal Components Analysis (IPCA), applying 1 Dimensional Convolutional Neural Networks (1D-CNNs) to extract spatial syntax, and Long Short-Term Memory (LSTM) networks to extract temporal behavioral characteristics. IPCA is used for online dimensionality reduction and still retains about 94% of the variance while significantly cutting down the number of features in the input layer of the deep learning architectures. To help approaches generalize better in “zero-day” attack scenarios, Generative adversarial Networks (GANs) were used on a couple of popular datasets, CIC-IDS2017 as well as CSE-CIC-IDS2018, to overcome the inherent class imbalance of both datasets. Experimental results were performed using an Azure Machine Learning environment that shows hybrid CNN-LSTM model has improved classification with 97.91% accuracy and 97.29% F1-score. Notably, using IPCA has brought the average inference time down to 42.5ms, which is over 30% faster than the baseline and data reduction compared approaches. This would be a promising framework to deploy modern deep learning based cyber defense solutions which are scalable and operationally deploy-able in today ‘s web systems without perceptibly affecting throughput.