NPUMeter: Automatic Operator Optimization for Ascend NPU with Accurate Analytical Performance Models

With the rapid development of AI and deep learning, computational demands are increasing significantly. While GPUs excel in parallel computing, they fall short in terms of energy efficiency, specialization, and processing latency. In contrast, Neural Processing Units (NPUs), such as the Ascend NPUs, designed specifically for deep learning tasks, demonstrate superior performance. However, the architecture specialization makes operator development more challenging, leading to a reliance on manual tuning and optimization, which incurs significant time cost and developing effort. To address this issue, we propose NPUMeter, an automatic operator optimization framework for Ascend NPUs built upon accurate and comprehensive analytical performance models. NPUMeter comprises two components: 1) an analytical performance model that accurately estimates operator latency on NPU given different configurations of optimization parameters; 2) an efficient design space exploration (DSE) algorithm that automatically searches for the optimal parameter configuration in a large design space within minutes. Experimental results demonstrate that NPUMeter achieves high estimation accuracy, with an average error below 5%. It effectively generates near-optimal configurations for various operators, achieving up to a 1.46 × performance speedup compared to the configuration generated by the Ascend C compiler while reducing the DSE time from hours to minutes.