3CBench: A Unified Benchmarking Framework for the Computing Capacity of Heterogeneous AI Clusters

Abstract The rapid evolution of Artificial Intelligence has driven the demand for extensivecomputational resources and the deployment of AI tasks across heterogeneouscomputing platforms. However, existing benchmarking systems face several chal-lenges, including limited compatibility with diverse hardware, insufficient supportfor varied deep learning frameworks and tasks, and a lack of comprehensive eval-uation metrics for the computing capacities. To address these issues, we propose3CBench, a unified benchmarking framework designed for heterogeneous AI clus-ters. Featuring a modular architecture encompassing environment management,task execution, and metrics analysis, 3CBench provides automated workflows andensures seamless compatibility with diverse GPU architectures and deep learningframeworks. It provides a comprehensive evaluation metrics system to rigorouslyassess computational performance and stability across both transformer-basedlarge language models and convolutional neural networks, thereby covering dom-inant deep learning architectures. Extensive experiments demonstrate 3CBench’sscalability on heterogeneous AI clusters, compatibility with various deep learningframeworks and tasks, and the support for a wide range of applications. Addition-ally, 3CBench aids in problem diagnosis during the development process of GPUvendors. These features establish 3CBench as a robust tool for benchmarking,optimization, and system-level evaluation in heterogeneous AI clusters.