FrBGNN: Accelerating GNN Training by Concurrent Backward Gradients Computations on GPU

Graph Neural Networks (GNNs) have gained significant traction due to their state-of-the-art performance on various graph-related tasks. However, GNN entails hybrid executions of irregular aggregation and regular update operators, which manifests complex operator dependencies and limited computational efficiency in the forward and backward propagation. Especially in backpropagation, advanced works compute gradients sequentially, which overlook the complex dependencies among gradient operators. We propose FrBGNN, a fused framework with operator dependency optimizations for concurrent GNN training. By exploiting the dependencies of gradient operators, FrBGNN first enables to reorder their execution to make the shortest backward dataflow. Then, a novel pull-to-push formulation is proposed to fuse the backward gradients along the execution path, which transforms the output-driven pull formulation to the data-driven push formulation. FrBGNN integrates concurrent multi-stream with asynchronous pipeline to reduce data movement and access overhead effectively. In the backward phase, experimental results demonstrate that FrBGNN achieves up to 5.2 × performance speedup over DGL. Moreover, FrBGNN promotes the overall performance by 2 to 6.1 × compared to existing GNN frameworks.