This is the authors' abstract. We don't add key points for this paper.
With the rapid increase in the size of neural networks, model compression has become an important area of research. Quantization is an effective technique for decreasing the size, memory access, and compute load of large models. In this paper, we first benchmark the impact of techniques such as straight through estimator, pseudo-quantization noise (PQN), learnable scale parameter, clipping, etc. on 4-bit seq 2 seq models across a suite of speech recognition datasets ranging from 1,000 hours to 1 million hours, as well as one machine translation dataset to illustrate its applicability outside of speech. It is commonly believed that having dedicated learnable scale parameters per quantization group is critical to model accuracy. Instead, we propose to construct the scale as a L<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> norm of a few of the largest outliers within the quantization group and regularizing that norm within the end-to-end optimization. This outperforms popular learnable scale and clipping methods without the need to introduce extra parameters. PQN-QAT shows a larger improvement under the proposed method, and it opens up the potential to exploit some of its other benefits: 1) training a single model that performs well in mixed precision and 2) improved generalization on long form speech.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Qiu et al. (2024) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: