Adaptive µ-law Gradient Quantization for Training MLPs and CNNs

This paper presents an adaptive µ-quasilogarithmic gradient quantization framework aimed at reducing memory and computational demands during deep neural network training. The approach employs a companding function with a dynamically adjusted µ parameter that adapts to the statistical properties of gradients. Two quantization strategies are developed: a switching µ-quantizer that toggles between low-bit uniform and high-bit quasilogarithmic modes based on gradient variance, and a hybrid µ-quantizer that statically applies uniform quantization to small gradients and quasilogarithmic to larger ones. Experiments on multiple-layer perceptron (MLP) and convolution neural network (CNN) models trained on CIFAR-10 show that both quantizers retain classification accuracy close to full-precision (FP32) baselines while significantly reducing gradient reconstruction error (RMSE). The hybrid variant consistently achieves better validation accuracy, lower RMSE, and faster convergence than the switching scheme. These results highlight the potential of hybrid quasilogarithmic quantization as an efficient and scalable solution for training deep models in memory or bandwidth constrained environments.