Inside the Cerebras Wafer-Scale Cluster

The compute and memory demands of Machine Learning (ML) have driven the industry to use clusters of thousands of GPUs to train state-of-the-art models. However, scaling performance on GPU clusters is inherently complex, requires many forms of parallelism, and often has poor scaling performance. The Cerebras Wafer-Scale Cluster provides ExaFLOP level compute while avoiding these limitations by using only data parallelism. The cluster is built around the CS-2 system with the Wafer-Scale Engine-2 (WSE-2) processor that enables even the largest models to run on a single chip without partitioning. This unique property allows the cluster architecture to decouple memory and compute. Model weights are stored in an external memory device, MemoryX, and compute scaling is enabled using a special broadcast-reduce fabric, SwarmX. This article describes the challenges of GPU scaling and details the internal design of the Cerebras Wafer-Scale Cluster architecture and its unique ability to overcome these challenges.