MixDNN: Scheduling Optimization of Distributed DNN Training in Heterogeneous GPU Environments

With the rapid advancement of deep neural networks (DNN), the scale of workloads continues to expand, exhibiting diverse training characteristics. Traditional scheduling frameworks for PyTorch and TensorFlow, such as Kubernetes, typically assume uniform training workloads across samples within a given task. Consequently, existing frameworks adopt static parallelism strategies throughout the entire training process, leading to suboptimal performance in heterogeneous distributed training environments. As model complexity and GPU heterogeneity grow, this rigidity results in inefficient resource utilization and degraded scalability. To address these challenges, we propose MixDNN, an event-driven rolling-horizon scheduler for distributed DNN models in heterogeneous GPU environments. First, MixDNN divides submitted jobs into two categories, hard-model and soft-model. Then, MixDNN associates utility with different resource requests to represent GPU computing performance. At each job-arrival or job-completion event, MixDNN selects execution configurations and planned start times for waiting jobs by maximizing overall utility subject to GPU-capacity constraints. Running jobs are not preempted, migrated, or reconfigured. In the evaluation, MixDNN reduces parallel training time by up to 11.45% compared to Kubernetes and up to 70% compared to static scheduling, especially for distributed DNN models in heterogeneous GPU environments.