What question did this study set out to answer?

The aim is to enhance multiclass anomaly detection in network traffic, particularly for rare attack types affected by class imbalance.

February 12, 2026Open Access

An Optimized Deep Learning Approach for Multiclass Anomaly Detection

Key Points

The aim is to enhance multiclass anomaly detection in network traffic, particularly for rare attack types affected by class imbalance.
Developed a hybrid framework combining unsupervised feature learning and supervised classification.
Utilized a denoising autoencoder for robust feature representation on normal traffic.
Employed isolation forest for generating statistical anomaly scores.
Fused complementary features for classification with LightGBM.
Applied SMOTE-ENN for balanced training and conducted rigorous validation procedures.
Achieved approximately 99% classification accuracy and macro-F1 performance exceeding 97%.
Exceptional detection of rare attack classes, with F1 scores of 99% for R2L and 100% for U2R.
Outperformed previous methods, achieving significantly better recall rates for rare classes.

Abstract

The increasing scale and imbalance of modern network traffic pose significant challenges for multi-class intrusion detection systems (IDSs), particularly in identifying rare attack types. Traditional intrusion detection approaches based on supervised classification or unsupervised anomaly detection often suffer from limited generalization under severe class imbalance, high-dimensional feature spaces, and noisy traffic, resulting in poor detection of minority attack classes. To address these limitations, this study presents a hybrid intrusion detection framework that integrates unsupervised feature learning, anomaly scoring, and supervised classification within a unified pipeline. A denoising autoencoder trained exclusively on normal traffic is employed to learn compact and noise-resistant feature representations, while an isolation forest independently generates statistical anomaly scores. These complementary features are then fused and classified using a Light Gradient Boosting Machine (LightGBM). The main contribution of this work lies in the effective integration of these components, combined with a balanced training strategy based on the Synthetic Minority Over-sampling Technique with Edited Nearest Neighbors (SMOTE-ENN), as well as robust validation procedures. The framework is evaluated on the Network Security Laboratory Knowledge Discovery and Data Mining dataset (NSL-KDD) and the UNSW-NB15 intrusion detection dataset using stratified cross-validation and multiple independent runs. Experimental results demonstrate consistently high classification accuracy (~99%) and strong macro-F1 performance (>97%) across all attack categories on both NSL-KDD and UNSW-NB15 datasets. The framework achieves exceptional detection of rare classes (R2L: 99% F1, U2R: 100% F1), significantly outperforming prior approaches (AE-SAC: 83.97% F1, RL-NIDS: poor U2R recall), while maintaining low inference latency (~2–3 ms per sample, 415 samples/second) suitable for real-time network security deployment.

Mark Helpful

Bookmark

Relay

View Full Paper