A comprehensive survey on multi-GPU systems

Graphics Processing Units (GPUs) play a critical role in contemporary applications that handle large data volumes. Nonetheless, the performance enhancement provided by GPUs is constrained by their memory capacity and bandwidth limitations, which can hinder their efficiency in processing extensive datasets. To address these limitations, multi-GPU architectures are increasingly recognized as an effective approach to meet the growing demands of modern parallel applications. These systems facilitate a scalable, high-performance computing environment by harnessing the processing capabilities of multiple GPUs. A multi-GPU configuration not only delivers remarkable computational throughput but also enhances memory bandwidth and capacity, which are crucial for efficiently managing extensive parallel data-intensive workloads. Multi-GPU systems provide substantial benefits for deep learning, scientific simulations, and large-scale data processing tasks that demand the efficient handling of terabytes of data. However, multi-GPU systems grapple with critical challenges, including inefficient memory management, communication bottlenecks, and uneven distribution of workloads. This paper presents a comprehensive review of recent software and hardware techniques addressing prevalent performance bottlenecks of multi-GPU systems. We specifically focus on methodologies optimizing cache management, improving load balancing, enhancing inter-GPU communication, and refining memory migration. Furthermore, the paper identifies emerging trends and unresolved challenges, offering insights into potential future directions for research and development. This comprehensive study is intended to serve as a foundational resource for both academics and industry professionals researching to advance the efficiency and scalability of multi-GPU systems.