A novel hyperparameter tuning method for enhanced intrusion detection in network security

Intrusion Detection Systems (IDS) are essential for ensuring the security of enterprise networks and cloud-based systems, as they defend against sophisticated and evolving cyberattacks. Machine learning (ML) techniques have emerged as effective tools to enhance IDS performance, addressing the limitations of traditional methods. This study proposes a novel hyperparameter tuning method for ML-based IDS, leveraging the NSL-KDD dataset with extensive feature selection and preprocessing to address data imbalance and redundancy. The method, integrating adaptive refinement with stochastic perturbation, optimizes classifiers such as Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB), achieving both higher detection accuracy (99.90% with RF) and improved computational efficiency. This approach excels due to its dynamic adjustment of parameter ranges and controlled randomness, converging faster than traditional Grid Search and Random Search by reducing iterations by up to 87.5%. The experimental results demonstrate that tree-based models, particularly RF, outperform others due to their ability to model complex, non-linear patterns, enhanced by the proposed tuning method. Measured in terms of convergence speed, CPU time, and memory usage, this method proves suitable for deployment in real-time, resource-constrained environments, offering a scalable and efficient solution for network security.