Mixed-precision quantization has shown great potential in reducing memory requirements and improving inference efficiency and has received extensive attention. However, narrowing bit lengths does not always translate into significant efficiency gains for various reasons. Firstly, computation kernels often lack support for mixed-precision operations, limiting their utility in systems that rely on such kernels. Secondly, as DNN computation involves multiple stages, the precision pattern can vary significantly across these stages. This complicates the development of precision-aware optimizations for achieving end-to-end efficiency gains. Ultimately, effective precision-aware optimization necessitates additional support throughout the entire vertical stack. This paper proposes Mix-PC, a framework that enables efficient mixed-precision compilation for high-performance DNN inference. On the one hand, we delve into the cause of inefficiencies in the current compilation for supporting mixed-precision operators. On the other hand, we propose Mix-PC, a general-purpose compilation framework that leverages adaptive patterns for different numeric precision. It leverages adaptive patterns tailored for different numeric precision and progressive fusion strategies, enabling enhanced compatibility with diverse models. The results show that compared to the SOTA quantization efforts, Mix-PC achieves 1.70 ×, 1.34 ×, and 2.13 × speedup, respectively, in BERT, ViT, and HuBERT.
Huang et al. (Fri,) studied this question.