Ciri: Scaling-aware job scheduling for small language models on over-committed GPU clusters

Despite the remarkable breakthroughs in large language models (LLMs), researchers are also designing models with modest size (hundreds of millions to billions of parameters) for domain science and agentic AI. Training and fine-tuning modest models on over-committed GPU clusters can experience long turnarounds due to the high computational demand and extensive queue wait times. Enabled by gradient accumulation, such a workload can leverage any number of GPUs as long as the global mini-batch size is fixed. Training with more GPUs can reduce execution time, but leads to longer queue wait times and higher node hour consumption due to nonlinear scaling efficiency. There is a clear tradeoff between turnaround time and the node hour consumption, which is agnostic to existing schedulers. In this paper, we present Ciri, a reinforcement learning-enabled and scaling-aware user-level job scheduler, which incorporates workload scaling curves and machine states into scheduling decisions. Ciri dynamically selects the GPU count to trade off turnaround time for node hour consumption. We evaluate Ciri with three GPT models: GPT-345M, GPT-1.5B, and GPT-2.0B with job traces of the TACC Lonestar6 and Frontera GPU clusters. The results show that Ciri can effectively reduce the node hour consumption by 0.9-1.8% for every 1% increase in turnaround across the two machines.