VAE‐Based SPC for Botnet Detection in IoT Systems

The rapid proliferation of Internet of Things (IoT) devices has created new security risks, particularly botnet infections. This paper proposes an unsupervised monitoring framework that combines a ‐variational autoencoder (VAE) with multivariate statistical process control (SPC) for botnet detection in IoT networks. Categorical fields of network flows are represented with per‐field embeddings, trained jointly with the VAE and concatenated with the scaled numeric features. The VAE is trained only on benign traffic; the resulting latent code and reconstruction residuals are monitored with Hotelling's and squared prediction errror (SPE) control charts, with control limits calibrated by a moving block bootstrap that preserves local temporal dependence of network flows during Phase I calibration. A per‐flow EWMA variant of both charts is also evaluated as an alternative to fixed‐length monitoring windows. On 12 IoT‐23 botnet families, the proposed charts reach pooled AUCs of 0.9906 () and 0.9946 (SPE) against held‐out benign traffic, with 100% flow‐level recall in every family. However, at the bootstrap‐calibrated operating point, the held‐out benign false‐positive rates are 75.3% and 75.1%, respectively, revealing a substantial distribution shift between the Phase I reference and the subsequent benign traffic. The framework is benchmarked against seven unsupervised baselines and four supervised classifiers. A component‐wise ablation shows that the learned embedding representation is a major contributor to AUC, whereas a deterministic autoencoder using the same embeddings achieves comparable AUC; a separate threshold‐sensitivity analysis evaluates the effect of the operating threshold. Quantitative latent‐space separability measures (silhouette and Davies–Bouldin) identify the botnet families that remain hardest to separate from benign traffic.