Large language model (LLM) serving has emerged as a major source of energy consumption in modern AI infrastructure. In current deployments, graphics processing units (GPUs) are typically operated at default high-frequency settings to maximize performance. However, under practical service-level objectives (SLOs), peak performance is often unnecessary, especially during the memory-bound decode stage, resulting in substantial power redundancy and avoidable energy waste. Existing studies that apply GPU dynamic voltage and frequency scaling (DVFS) to improve the energy efficiency of LLM serving have shown promising results. However, they generally rely on coarse-grained control, accurate output length prediction, or request-level resource management, which limits their effectiveness under highly dynamic workloads and strict SLO constraints. We present EcoInfer, a fine-grained DVFS framework for energy-efficient LLM serving. EcoInfer performs iteration-level, workload-aware GPU frequency control that adapts to the current inference phase and system state while preserving latency guarantees. It comprises three tightly integrated modules: a machine-learning-based frequency–latency predictor that estimates iteration latency across candidate GPU frequencies using lightweight iteration-level features; an SLO-aware frequency controller that selects the minimum feasible frequency within a sweet-spot-guided candidate range; and a low-overhead runtime optimization layer that combines adaptive decision caching with asynchronous execution to reduce and hide the overhead of online control. Implemented on top of vLLM, EcoInfer achieves up to 25.4% energy savings and 21.5% average energy savings and improves energy efficiency by 1.28× on average in terms of Tokens/J while maintaining a nearly unchanged SLO attainment rate compared with the default vLLM baseline.
Hu et al. (Sat,) studied this question.