The Utilization Fallacy and the Real Drivers of Carbon-Efficient Inference Serving

Cloud providers deploy massive fleets of GPUs to meet the growing demand for machine learning inference, but these fleets come with a steep carbon cost—manufacturing 350,000 NVIDIA A100 GPUs emits an estimated 7.54 million kgCO2. Prior efforts have largely focused on increasing GPU utilization, under the assumption that higher utilization translates to better carbon efficiency. However, we find the notion that higher GPU utilization makes inference serving systems inherently carbon efficient to be a fallacy. Through our characterization study, focusing on the carbon efficiency of GPU spatial sharing, we find that optimizing for resource utilization does not always achieve carbon efficiency; the outcome depends on the specific models co-located on a GPU. This tradeoff between utilization and carbon efficiency is shaped by multiple drivers, including fluctuations in the underlying energy sources, request rates, and model input/output requirements. Thus, improving the sustainability of inference serving demands a shift from utilization-focused designs to carbon-aware GPU sharing and runtime policies. To realize our vision, we (1) introduce an efficient and accurate preliminary methodology to estimate GPU power consumption under concurrent model execution, and (2) show that frequency tuning in shared GPUs can be used as a lever to improve carbon efficiency, but must be tailored to the combination of models sharing a GPU and key carbon-efficiency drivers, brought up by our characterization study. We conclude by proposing new avenues for research as next steps and a call to action for the hardware community to improve the long-term sustainability of ML inference serving.