HeteroGEMM: Searching for Faster, Heterogeneous GEMM Accelerators for DNNs

This is the authors' abstract. We don't add key points for this paper.

The General Matrix Multiply (GEMM) operation underpins several fundamental building blocks of deep neural networks (DNNs), including convolutional, dense, recurrent, and attention layers. Traditional FPGA-based accelerators utilize a single systolic array kernel to perform GEMM. However, the diversity in sizes of GEMMs utilized by various algorithms, combined with the inter-dependencies between different DNN layers render a monolithic kernel with fixed configuration inefficient for diverse computational patterns. We propose HeteroGEMM, an open-source framework for searching heterogeneous multikernel accelerators tailored to GEMM computational graphs of DNNs. HeteroGEMM leverages GEMM algorithms-such as the Strassen's algorithm-alongside adaptable systolic array configurations and a range of other tunable parameters to search for a combination of GEMM kernels that maximize system throughput under multi-die FPGA resource constraints on user-defined GEMM computational graphs. We perform a suite of experiments on GEMM computational graphs obtained from diverse models including ViT, BERT, ResNet18, and an MLP on two high-performance FPGA platforms. The results demonstrate that searching for bespoke multi-kernel accelerators yields significant performance benefits compared to prêt-à-porter expert-designed multi-kernel accelerators.