What question did this study set out to answer?

This research aims to enhance quantization methods for large language models by employing layer-specific strategies.

December 22, 20250 citationsOpen Access

CKA-Guided Modular Quantization: Beyond Bit-Width to Algorithmic Diversity

JZJinhao ZhangChongqing Normal University YZYunquan ZhangChinese Academy of Sciences DCDaning ChenGuangxi University

Key Points

This research aims to enhance quantization methods for large language models by employing layer-specific strategies.
Introduces CKA Guided Modular Quantization framework for heterogeneous algorithm selection
Utilizes Linear Centered Kernel Alignment (CKA) to evaluate quantization algorithms per layer
Integrates optimized quantization strategies to create a hybrid model
Demonstrates improved performance over uniform quantization and mixed-precision methods
Achieves lower perplexity in tests with mainstream language models like LLaMA and Qwen

Abstract

Current mainstream post-training quantization methods for large language models typically apply a uniform quantization strategy across all network layers, overlooking the substantial differences in algorithmic suitability among layers. To address this limitation, we propose CKA Guided Modular Quantization, a fine-tuning-free, plug-and-play framework for algorithmic heterogeneous quantization. Our method independently evaluates multiple PTQ algorithms on each layer and employs Linear Centered Kernel Alignment (CKA) as a metric to automatically select the optimal quantization strategy per layer. The individually optimized strategies are then integrated to construct a hybrid quantized model. Experiments demonstrate that our approach consistently outperforms both uniform quantization baselines and state-of-the-art mixed-precision methods across mainstream LLMs including LLaMA and Qwen ,in terms of perplexity (PPL) and downstream task performance.

Ask AI

Helpful

Bookmark

View Full Paper