Enhanced multiclass intrusion detection on imbalanced IoT NetFlow traffic using a three-tier stacked time-distributed LSTM with SMOTENC-based class balancing

The network-based attack surface has flourished abruptly with the unveiling of Internet of Things (IoT) devices. As a result, there is a pressing demand for automated intrusion detection systems (IDS) that can reliably identify a variety of attacks with severe class imbalance. Deep recurrent models are suitable for NetFlow traffic, however, a single-layer long short-term memory (LSTM) network is often inadequate for separating minority attack classes, and the effect of oversampling and model size is not well quantified. This study presents a three-tier stacked Time-Distributed long short-term memory network (ST-LSTM) evaluated on NF-BoT-IoT (600,100 flows, four attack classes and one benign class) under a leakage-free protocol. Ten flow features serve as predictors, while the class label and host identifiers are excluded. The data are divided into training, validation and test partitions before any fitting, and mixed-type oversampling (SMOTENC) is applied to the training partition only, at a ratio selected on validation data. On the untouched test set of 120,020 flows, the model reaches 83.98% accuracy, 0.8500 weighted F1, 0.6609 macro F1 and 0.9678 weighted ROC-AUC, the highest values among classical, shallow recurrent and stacked-ensemble baselines, and five-fold cross-validation gives 83.94 ± 0.19% accuracy. Standard SMOTE, by contrast, creates invalid protocol and flag codes and suppresses DDoS detection. A label-conflict analysis shows that 99.9% of DDoS and DoS flows share identical feature vectors with other classes, which limits attainable five-class accuracy to 84.91%, the model reaches 98.9% of this bound, and nearly all of its remaining errors fall on these ambiguous flows.