On the impact of intra- and inter-node communication in the performance of interconnection networks in HPC and AI systems

Abstract In the last decade, specialized computing and storage devices, such as GPUs, TPUs, and high-speed storage, have been incorporated into server nodes of HPC and AI systems. The development of high-bandwidth memory (HBM) enabled a much more compact form factor for these devices, allowing the interconnection of several devices within a server node, typically using an intra-node interconnection network (e.g., PCIe, NVLink, or infinity fabric). Intra-node networks must allow efficient scale-up of the number of these devices within (and even beyond) a single node. Similarly, inter-node networks must enable efficient communication among hundreds of thousands of devices across thousands of server nodes when scale-up domains cannot be larger. Unfortunately, the intra- and inter-node networks may become the system’s bottleneck as communication demand among accelerators increases, driven by emerging applications such as generative AI. Although current intra-node network designs alleviate this bottleneck by increasing intra-node network bandwidth, intra-node communication may hinder inter-node communication performance when traffic from outside the server node arrives at the intra-node network. To evaluate the impact of this interference, we have analyzed intra- and inter-node communication operations generated under realistic traffic scenarios. We have developed a generic intra- and inter-node simulation model in OMNeT++ and have modeled the representative communication operations. We have also performed extensive simulation experiments confirming that increasing the intra-node network bandwidth and the number of computing devices per node (i.e., accelerators) may be counterproductive to the inter-node communication performance.