An Adaptive Quantization Method for CNN Activations

The post-training compression based on affine quantization is a common technology to improve the efficiency of embedded neural network accelerators. Current state-of-the-art quantization schemes for CNN activations usually rely on calibration dataset with better data representation to reduce the possibility of quantization overflow in inference, which is not always effective due to the uncertainty of the inference input data in practice. This paper proposes an adaptive quantization method for activations and its hardware-friendly design to directly address the quantization overflow in inference. This method monitors the quantization overflow of activations on-the-fly, adaptively updates the quantization parameters, and re-quantizes the activations when the overflow degree is over a tunable threshold. We evaluate the proposed method on VGG16 and MobileNetV2 and experiment results show that 11% improvement in inference accuracy at severe quantization overflow is achieved with the cost of 3% increase in runtime, and tuning the threshold enables trade-off between inference accuracy and speed.