Performance Prediction Techniques for Deep Learning Systems: A Survey

The rapid advancement of deep learning models has led to their widespread application across various domains; however, this has resulted in a significant increase in both hardware and software costs, necessitating optimizations for resource efficiency. To address this issue, performance prediction techniques evaluate the performance impact of corresponding optimizations on deep learning systems to maximize performance gains. This paper presents a comprehensive survey of performance prediction techniques for deep learning systems, categorizing them into allocation, cluster, and accelerator levels based on the system level at which their corresponding optimizations are situated, ranging from high to low. Allocation-level techniques primarily construct the relationships between resource configurations and their performance, cluster-level techniques model the optimizations within deep learning frameworks at given configurations, while accelerator-level techniques assess the efficiency of specific hardware. We point out that prediction techniques at different levels exhibit a hierarchical relationship rather than operating in isolation. High-level prediction techniques can incorporate the results from low-level techniques into their performance models to accommodate more low-level optimizations and improve their prediction accuracy. However, this will slow down the construction and execution speed of performance models, requiring trade-offs for specific use cases. Finally, we discuss future challenges and research directions in performance prediction techniques for deep learning systems.