Post-Training Weight Quantization in Small Language Models: Reasoning–Efficiency Trade-offs Across GGUF Precision Levels

Weight quantization can reduce the storage and computational requirements of languagemodels, but it may also affect their reasoning capabilities. This study investigates whetherquantization causes gradual or abrupt degradation in the reasoning performance of smalllanguage models. Three instruction-tuned models were evaluated: Qwen3-0.6B, Gemma 3 1BIT, Llama 3.2 1B Instruct. Each model was tested using a BF16 reference version and fiveGGUF quantization formats: Q8_0, Q6_K, Q5_K_M, Q4_K_M and Q3_K_M.Evaluation was performed on the same fixed 50-question subset of the GSM8K mathematicalreasoning dataset. Exact-answer accuracy, average latency, peak RAM usage, generationspeed, and stored model size were recorded. The results show that quantization did notproduce one universal degradation pattern. Accuracy changes varied across models andprecision levels. Qwen Q4_K_M matched its BF16 accuracy, Gemma Q4_K_M recorded a numerically higher accuracy than BF16 in this pilot sample, 44% versus 42%; this small difference should not beinterpreted as evidence of a meaningful improvement, and Llama Q6_K achieved the highest quantized accuracy for that model. However, Q3_K_M produced larger accuracy reductions for Qwen and Llama.Quantization consistently improved storage and inference efficiency. These findings suggestthat quantization provides practical efficiency benefits, but the effect on reasoningperformance is model-dependent.Keywords: weight quantization; small language models; GGUF; reasoning; llama.cpp;inference efficiency, GSM8K