Pipe-Torch: Pipeline-Based Distributed Deep Learning in a GPU Cluster with Heterogeneous Networking

Because training a deep neural network (DNN) takes arduous amounts of time and computation, often researchers expedite the training process via distributed parallel training on GPUs. On one hand, this lower computing-to-communication ratio makes traditional data parallelism difficult to scale, and traditional model parallelism leads to low GPU utilization. Both make it difficult to obtain a higher speedup. On the other hand, multi-GPU systems exhibit complex connectivity among GPUs. Overall, workload schedulers must consider hardware topology and workload communication requirements to allocate GPU resources for optimal execution time and improved utilization in GPU clusters with heterogeneous networking. Thus, in this paper, we introduce Pipe-torch, an improved pipeline-hybrid parallelism method (using both data and model parallelism) in a heterogeneous network environment. Ultimately, the framework's model partition algorithm aims to expedite pipeline-hybrid parallelism training between heterogeneous network-connected GPUs. Experiments with four different DNN models show that Pipetorch averages 1.4x speedup compared to data parallelism.