Toward Robust IDS in Network Security: Handling Class Imbalance With Deep Hybrid Architectures

The growing sophistication of cyberattacks demands Intrusion Detection Systems (IDS) that are both accurate and adaptive to diverse network threats. Traditional IDS often suffer degraded performance due to high-dimensional features and severe class imbalance in network traffic datasets. To address these issues, we propose a hybrid IDS framework integrating four optimized models (XGBoost, Long Short-Term Memory, MiniVGGNet, and AlexNet) enhanced through Random Forest Regressor-based feature selection and the Difficult Set Sampling Technique (DSSTE) for class balancing. Two integration strategies are employed: a hard-voting Ensemble and a Mixture of Experts (MoE) with a gating network for adaptive weighting. Comprehensive hyperparameter tuning via Keras Tuner and RandomizedSearchCV maximizes model performance. Evaluated on the CICIDS-2017 dataset, the system achieves detection rates above 99% with micro-average AUC values near 1.0, demonstrating strong generalization and effectiveness in detecting both majority and minority intrusions. The proposed framework holds strong relevance for security-critical domains, particularly wireless health monitoring systems, where ensuring the confidentiality and integrity of sensitive data is vital, thereby underscoring its suitability for real-world deployment.