What type of study is this?

This is a Quantitative Study study.

October 13, 2025Open Access

KurTail : Kurtosis-based LLM Quantization

MAMohammad Sadegh AkhondzadehHelmholtz Center for Information Security ABAleksandar BojchevskiUniversity of Cologne EEEvangelos EleftheriouIBM (United States)

Key Points

KurTail improves quantization by optimizing the measure of tailedness, effective in 4-bit quantization.
The method boosts MMLU accuracy by 13.3% and decreases Wiki perplexity by 15.5% compared to previous methods.
Layer-wise optimization ensures that weights, activations, and KV cache are efficiently quantized in 4 bits.
KurTail significantly lowers GPU requirements, needing only one GPU versus four for alternative approaches.

Abstract

One of the challenges of quantizing a large language model (LLM) is the presence of outliers. Outliers often make uniform quantization schemes less effective, particularly in extreme cases such as 4-bit quantization. We introduce KurTail, a new post-training quantization (PTQ) scheme that leverages Kurtosis-based rotation to mitigate outliers in the activations of LLMs. Our method optimizes Kurtosis as a measure of tailedness. This approach enables the quantization of weights, activations, and the KV cache in 4 bits. We utilize layer-wise optimization, ensuring memory efficiency. KurTail outperforms existing quantization methods, offering a 13.3\% boost in MMLU accuracy and a 15.5\% drop in Wiki perplexity compared to QuaRot. It also outperforms SpinQuant with a 2.6\% MMLU gain and reduces perplexity by 2.9\%, all while reducing the training cost. For comparison, learning the rotation using SpinQuant for Llama3-70B requires at least four NVIDIA H100 80GB GPUs, whereas our method requires only a single GPU, making it a more accessible solution for consumer GPU.

Demander à l'IA

Bookmark

View Full Paper