Bidirectional Centered Score Matching for Discrete Diffusion Distillation

Discrete diffusion models with parallel decoding are often distilled using token-wise KL objectives. Although token-wise KL exactly matches the corresponding product-form block distributions, its probability-space optimization geometry and teacher-only candidate truncation may provide insufficient supervision under aggressive few-step distillation. Concrete Score Matching (CSM) provides an alternative within-position objective by aligning relative logit geometry, but its complete pairwise form incurs O(|V|²) complexity. We propose Bidirectional Centered Score Matching (BCSM), an efficient within-position distillation objective that constructs an O(K) bidirectional active set, reduces pairwise comparisons to O(K²), and factorizes them into an O(K) objective. We further introduce cross-timestep transition alignment. Experiments demonstrate strong quality--efficiency trade-offs, including up to 6× NFE reduction with a 16.9% relative HumanEval gain and a 15.1% relative MMMU gain at single-step inference.