What Does MLX 4-bit Cost? A Controlled Audit of Quantization for Code Generation on Apple Silicon

Q: What does this research mean for the field?

MLX default 4-bit quantization significantly degrades code generation performance compared to bf16 baselines, a loss that shrinks with larger model sizes and cannot be recovered by calibrated AWQ. Novelty: ClaimNovelty.INCREMENTAL. Consensus alignment: ConsensusAlignment.NEUTRAL.

J. Melton

doi:10.5281/zenodo.22740849

September 14, 2026Open Access

What Does MLX 4-bit Cost? A Controlled Audit of Quantization for Code Generation on Apple Silicon

View Full Paper

Ask AI

Bookmark

Share

Key Points

To measure the performance trade-offs of default MLX 4-bit quantization against baseline bf16 representations in code generation models on Apple Silicon.
Evaluated five code generation models on HumanEval+ and MBPP+ benchmarks, comparing default MLX 4-bit quantization and calibrated AWQ directly against unquantized bf16 baselines converted from identical weights.
Performed pooled statistical discordance analyses using the McNemar test across evaluation pairs.
Audited evaluation harness integrity and examined the validity of benchmarking against published vendor figures.
Pooled results revealed 220 discordant pairs favoring bf16 compared to 125 favoring 4-bit quantization (McNemar p < 1e-6).
Performance degradation shrank systematically with increasing model capacity at a rate of +0.52 points per billion parameters, though a calibrated AWQ arm failed to recover the loss.
Identified five distinct benchmark harness defects that incorrectly penalized and scored valid model outputs as failures.

Ask about this paper

Answers are grounded in this paper and cite their sources.

Implication

Controlled audit demonstrates significant accuracy loss from MLX 4-bit quantization across code models, indicating that compression impairs performance despite parameter scaling.

Abstract

Controlled measurement of what MLX default 4-bit quantization costs five code models on HumanEval+ and MBPP+ against bf16 baselines converted from the same weights. Pooled: 220 discordant pairs favour bf16 against 125 (McNemar p < 1e-6); the loss shrinks with model size at +0.52 points per billion parameters. A calibrated AWQ arm does not recover it. Also documents why differencing against a vendor's published figure is unsound, and five harness defects that scored correct answers as failures.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

J. Melton (2026) studied this question.

synapsesocial.com/papers/6aa913819013453be30a17ad https://doi.org/10.5281/zenodo.22740849

View Full Paper

Ask AI

Bookmark

Share

What Does MLX 4-bit Cost? A Controlled Audit of Quantization for Code Generation on Apple Silicon

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

What Does MLX 4-bit Cost? A Controlled Audit of Quantization for Code Generation on Apple Silicon

Key Points

Ask about this paper

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading