A Thermal-Aware Workload Scheduler for High-Performance LLM Inference in Cooling-Regulated Datacenters

The wide deployment of AI datacenters has led to an increasing demand for energy. Recently, there have been developments in the area of reference cooling temperatures in datacenter server rooms, with the aim of avoiding the setting of excessively low temperatures to save energy. For example, the recommendation in Singapore is [28°C–32°C] for level 4 datacenters, while the European Union code of conduct for datacenters suggests a temperature of 35°C. These standards were carefully tested to satisfy overall cooling requirements, yet we observe in this paper that there is a risk that, in certain scenarios, the heat that is dissipated may not match the heat generation of GPUs, especially for high-performance workloads such as LLM inference. Such a mismatch can lead to an increase in the temperature of the GPU and trigger its thermal-throttle mechanism. The GPU frequency will decrease in self-protection from the damage due to overheating and performance degradation. As such, the issue of cooling regulation poses challenges to high-performance computing in AI datacenters. In this paper, we study LLM inference serving in cooling-regulated datacenters. Specifically, a datacenter serves millions of LLM inference jobs. To maximize the throughput, a workload scheduler (e.g., Ray Serve) assigns the jobs to GPUs and determines the execution batch sizes on GPUs. We show that in cooling-regulated datacenters, existing schedulers can increase the probability of thermal throttling by 10 times, and the performance degradation can be as much as 34.2%. We develop a new thermal-aware workload scheduler, TAWS, which takes into consideration the GPU voltage and frequency. Our scheduler can maximize the throughput of LLM inference under a relatively high ambient temperature in datacenter server rooms. The evaluation results show that the new scheduler can lead to a maximum improvement of 40.94% of throughput under 41°C.