Answers are grounded in this paper and cite their sources.
Quantizing a model's weights to int8 is widely assumed to make CPU inference faster as well as smaller. This study pre-registered that assumption and measured it in an engine built from scratch for the purpose: a C++20 transformer inference engine (own safetensors parser, tensor kernels, KV cache, and quantization path; standard library only; no inference library linked) running GPT-2 small (about 124M parameters) on one named machine (AMD EPYC 9D25, 4 vCPUs). The int8 arm is weight-only symmetric int8 with per-output-channel scales over the 48 block linear matrices (W8A32: weights int8, activations fp32, dequantize-on-the-fly in the kernels); embeddings, biases, and LayerNorm parameters stay fp32 in both arms. Three hypotheses were frozen in advance with exact decision rules. H1 (speed) is REFUTED: int8 generation throughput is 3.395 tokens per second against the fp32 baseline's 5.713 (ratio 0.594), far under the registered 1.50 line; the dequantize-on-the-fly inner loop costs more than the memory traffic it saves under this build. H2 (quality cost) is CONFIRMED with room to spare: perplexity on a frozen 8,192-token WikiText-2 test slice is 33.8158 (int8) against 33.8165 (fp32), a relative change of minus 0.002 percent, against a registered 5 percent bound. H3 (memory) is CONFIRMED: peak memory (VmHWM) is 310.9 MiB against 567.0 MiB (ratio 0.548, registered line 0.60), and weight storage falls from 497,759,232 to 243,287,040 bytes. A 12-prompt degeneration suite (descriptive, no hypothesis) shows both arms looping heavily under greedy decoding (mean rep20 0.954 fp32, 0.979 int8) and staying sane on all 12 prompts. Correctness was gated before benchmarking: 14 automated tests pass, including a tiny end-to-end fixture whose fp32 logits match an independent float64 NumPy reference within 4.2e-07, and a real-model cross-check reproducing the engine's block-0 perplexity to five significant figures. The claims are deliberately narrow: one small model, one machine, one quantization scheme, greedy decoding. Within those bounds, int8 bought memory and cost speed, and the quality cost was effectively zero.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: