Comprehensive survey reveals bottlenecks across drafting, verification, and execution in speculative decoding, highlighting trade-offs that limit large language model acceleration.
Key Points
To identify and analyze the fundamental system trade-offs and pipeline bottlenecks that constrain the real-world acceleration of speculative decoding in large language models.
Synthesized existing speculative decoding literature into a unified analytical framework.
Evaluated the interdependent contributions of draft model generation, parallel verification schemes, and hardware execution constraints.
Finds that isolated optimizations in draft prediction or verification mechanisms fail to yield expected speedups without holistic alignment across the entire inference pipeline.
Identifies that end-to-end acceleration is bounded by hardware execution overheads and the trade-off between draft model acceptance rate and verification latency.