Closed-Loop Target-Informed Speculative Scheduling for Accelerated Edge LLM Inference

CT-Spec: Closed-Loop Target-Informed Speculative Scheduling for Accelerated Edge LLM Inference Speculative decoding is a key acceleration strategy to bypass the memory-bandwidth bottleneck of autoregressive Large Language Model (LLM) inference by interleaving fast draft generation with parallel target verification. However, production inference engines—such as llama.cpp and vLLM—rely on a static speculative draft horizon (n_draft). This introduces an acute efficiency dilemma: shallow horizons starve target compute during structured syntax, while deep horizons cause severe rejection collapse and latency penalties on high-entropy branching text (dropping acceptance down to 75.8%). Prior adaptive approaches rely on open-loop draft-model confidence, suffering from draft blindness where small models remain confidently misaligned with the target distribution. CT-Spec establishes a target-informed closed feedback loop: it dynamically monitors ground-truth target verification acceptance rates (alpha_t) and target logit entropy (H_t) to modulate the speculative horizon in real time (n_draft in [1, 16]) with zero per-token drafting overhead.