Towards high scalability and fine-grained parallelism on distributed HPC platforms

Current High-Performance Computing systems rely on massive parallelism to achieve exascale performance. They use task scheduling and message-passing programming models to explore complementary sources of parallelism. Combining the two holds the promise of allowing seamless exploitation of both intra- and inter-node concurrency while leveraging widely-known programming abstractions. Still, the interaction between the two raises coordination problems that could make work distribution excessively costly, limiting performance. This work is the first to evaluate comprehensive hardware acceleration of their combined use integrating them in a programming model that further exploits their synergies. The hardware/software co-design approach proposed for this purpose is prototyped on a cluster of 64 FPGA nodes, where each holds a RISC-V Rocket Chip CPU with 8 cores. On one hand, this paper combines OMPIF and Picos, which are hardware accelerators for message passing and task scheduling respectively. They interface with the CPU through RoCC-based custom RISC-V instructions. On the other hand, we present the Implicit Message Passing (IMP) programming model, that extends task scheduling abstractions to leverage MPI-mediated inter-node parallelism without requiring explicit MPI calls. Thus, IMP transparently allows the dataflow-style execution induced by task scheduling to span multiple nodes. We implement three benchmarks, N-body, Heat, and Cholesky, each with two different strategies, IMP and explicit MPI, and evaluate them on the multi-core FPGA-based cluster. We demonstrate our hardware-software co-design approach achieves near-linear scalability with IMP and the OMPIF/Picos accelerators, and reduces task management overhead from 2200 to 300 cycles per task. Furthermore, when leveraging all 512 cores (split among the 64 nodes), we measure speedups of 2.04x (40x in communication), 1.25x (7x in communication), and 7.29x (25x in communication) compared to unaccelerated MPI for N-body, Heat, and Cholesky respectively. Finally, at 64 nodes, we respectively achieve 99%, 83%, and 79% of weak scaling efficiency.