Automated Mixed-Precision Weight-Only Quantization optimizes model quality and memory usage, suggesting new pathways for deploying LLMs under constraints.
Key Points
AMQ achieves high-performance large language models by balancing memory usage and model quality efficiently.
The framework explores over 10^{100} configurations, employing search space pruning to enhance performance.
Innovative components like quantization proxy and quality predictor minimize computational overhead during optimization.
Iterative search-and-update strategies ensure fast convergence towards the Pareto frontier of quality and efficiency.