GNConvNet: a group normalization convolutional neural network for robust and secure channel estimation in next-generation OFDM systems

Channel estimation in Orthogonal Frequency Division Multiplexing (OFDM) systems remains a critical challenge in fifth-generation (5G) and sixth-generation (6G) wireless networks, particularly under adversarial perturbations that can severely degrade the reliability of deep-learning-based estimators. Conventional normalisation techniques, notably Batch Normalization (BN), exhibit well-documented instability under small or variable batch sizes and limited robustness to adversarial inputs, thereby restricting their suitability for security-sensitive edge deployment scenarios. This paper proposes GNConvNet, a compact fully-convolutional channel estimator in which BN is replaced by Group Normalization (GN), and which is trained within a teacher–student defensive distillation framework that combines a Kullback–Leibler distillation loss, a clean-data mean-squared-error (MSE) loss, and an adversarial-MSE term computed on Fast Gradient Sign Method (FGSM) perturbations. The model is evaluated on the standard MATLAB 5G Toolbox dataset (512 channel realisations under TDL-A through TDL-E profiles with SNR 0–10 dB) in a SISO downlink OFDM configuration, matching the foundational study by Catak et al., against four white-box attacks (FGSM, BIM, MIM, PGD) at perturbation budgets ε ∈ {0.1, 0.5, 1.0, 2.0, 3.0}. To eliminate hyperparameter-selection leakage, the 32-sample MATLAB validation set is deterministically partitioned into a 16-sample development-validation subset (for hyperparameter tuning) and a 16-sample held-out test subset (for final reported results). A six-point batch-size sweep over B ∈ {2, 4, 8, 16, 32, 64} confirms that GN outperforms BN at every small-batch setting, with the largest gap (60%) at B = 16. A four-way ablation on the held-out test set isolates the contributions of GN (9.1% MSE reduction) and Distillation (7.0%), with the combined GNConvNet achieving a 13.1% total mean-MSE reduction and a 9 × reduction in inter-seed variance relative to the BN baseline. Across 15 matched-seed trials, GNConvNet reaches a mean test MSE of 0.021534 ± 0.000305 versus 0.025571 ± 0.003024 for BN (Welch’s t = − 7.34, p = 1.07 × 10 −6 ; Wilcoxon p = 6.10 × 10 − 5; Cohen’s d = 2.68; GN wins 15 of 15 trials). Under adversarial attack at ε = 3.0, GNConvNet achieves 5–58 × lower malicious MSE than the BN baseline across all four attack families. The proposed model is compact (92,641 trainable parameters, 16 ms single-sample CPU inference) and therefore well-suited to real-time edge deployment in next-generation base stations. The complete reproduction notebook implementing the deterministic split and all experiments is publicly released.