Evaluation Pitfalls in Post-Training Quantization of Diffusion Transformers: An Empirical Audit

This is the authors' abstract. We don't add key points for this paper.

Diffusion transformers are moving toward practical deployment, and post-training quantization (PTQ) has emerged as a standard, retraining-free compression route. The conventions used to evaluate PTQ for diffusion transformers—the Fréchet Inception Distance (FID) of images produced by a fixed sampling pipeline, with a cheap calibration-error statistic as a configuration-search proxy—are typically inherited from prior codebases rather than audited. Auditing this pipeline through a DiT-XL/2 (ImageNet 256×256) case study of 209 controlled experimental records, this paper identifies three systematic pitfalls. First, sampling-pipeline configuration: two latent defects in the legacy pipeline manufactured spurious method-level differences, and controlled A/B reproduction (deviation below 1%) shows that correcting them reverses earlier conclusions on channel equalization. Second, proxy-metric failure: 32 records sharing a bit-identical calibration error span an FID range of 98.74, and its rank correlation with FID is statistically indistinguishable from zero in every experiment family tested. Third, calibration-protocol sensitivity: under the corrected protocol, an error-aware dynamic-programming grouping strategy is statistically indistinguishable from a fixed-interval baseline (273.62±1.78 vs. 274.32±1.80), while group-wise equalization turns from apparently beneficial to harmful (273.62→313.23). The only method-level conclusion that survives the protocol correction—under three seeds, a single model, and the 8-bit weight-and-activation (W8A8) configuration studied—is that grouped quantization outperforms no grouping (388.96→274.32). This work closes with a corrected evaluation-protocol checklist.