What question did this study set out to answer?

This research aims to evaluate federated learning-based intrusion detection systems in IoT networks using diverse datasets to enhance generalization and performance.

January 18, 2026Open Access

Dataset-centric evaluation of federated intrusion detection models in IoT networks

Key Points

This research aims to evaluate federated learning-based intrusion detection systems in IoT networks using diverse datasets to enhance generalization and performance.
Evaluated federated learning models using three IoT datasets: Edge-IIoTset, CIC-IoT2023, and TII-SSRC-23.
Benchmarking of three federated learning aggregation algorithms: FedAvg, FedProx, and FedNova.
Assessment of models using two deep learning architectures: LSTM and Transformer.
Constructed non-IID client scenarios based on device/application types and aligned feature sets to harmonized labels.
Conducted evaluations across in-domain, cross-dataset, and combined multi-dataset settings.
Federated models achieved near-centralized performance in-domain, with macro-F1 scores up to 98% and accuracies between 92% and 98%.
Transformers outperformed LSTMs by about 1-2 percentage points in macro-F1 scores while maintaining comparable communication costs.
Cross-dataset tests revealed significant performance drops of up to 30 percentage points in macro-F1 when exposed to unseen environments.
Multi-dataset federation restored transfer, achieving around 90% macro-F1 across different datasets.
FedProx improved stability and reduced variance between training rounds, while FedNova required fewer rounds to achieve target accuracy and reduced communication costs by approximately 15-25% compared to FedAvg.

Abstract

Abstract Intrusion detection systems (IDS) leveraging federated learning (FL) are increasingly deployed in Internet of Things (IoT) environments to address distributed data and privacy constraints. However, generalization remains unclear because most evaluations rely on a single dataset, which risks overfitting to site-specific traffic, label taxonomies, and non-IID client mixtures. This study provides a comprehensive dataset-centric evaluation of FL-based IDS across three contemporary IoT/IIoT datasets: Edge-IIoTset (2022), CIC-IoT2023, and TII-SSRC-23 (2023), that differ in devices, feature distributions, and attack families. We benchmark three FL aggregation algorithms (FedAvg, FedProx, FedNova) with two deep learning backbones (LSTM and Transformer) to assess detection accuracy, cross-environment generalizability, convergence behavior, and communication cost. Methodologically, we construct non-IID clients by device or application type, harmonize labels to a common family-level schema, align features to the intersection set, and evaluate three regimes: in-domain, cross-dataset, and a combined multi-dataset federation. Results show that federated models approach centralized performance in-domain, with macro-F1 up to 98% and accuracies in the 92–98% range. Transformers consistently exceed LSTM by 1–2% points in macro-F1 at comparable communication budgets. Cross-dataset tests expose substantial degradation, with up to 30 percentage-point macro-F1 loss when models face unseen environments, underscoring the need for diverse training coverage. Combined multi-dataset federation substantially restores transfer, yielding 90% macro-F1 across datasets in the harmonized family-level setting. Under heterogeneous clients, FedProx improves stability by reducing round-to-round variance, while FedNova achieves target accuracy in fewer rounds and lowers communication by 15–25% relative to FedAvg. These findings indicate a practical recipe for deployment: prioritize attack and environment diversity through combined-dataset FL, select Transformer backbones where feasible, and use FedProx or FedNova to stabilize training and reduce communication in bandwidth-constrained IoT settings.

Dataset-centric evaluation of federated intrusion detection models in IoT networks

Key Points

Abstract

Cite This Study