Accelerating Zero-Shot NAS With Feature Map-Based Proxy and Operation Scoring Function

This is the authors' abstract. We don't add key points for this paper.

Neural Architecture Search (NAS) has been extensively studied due to its ability in automatic architecture engineering. Existing NAS methods rely heavily on the gradients and data labels, which either incur immense computational costs or suffer from discretization discrepancy due to the supernet structure. Moreover, the majority of them are limited in generating diverse architectures. To alleviate these issues, in this paper, we propose a novel zero-cost proxy called MeCoMeCo based on the Pearson correlation matrix of the feature maps. Unlike the previous work, the computation of MeCoMeCo as well as its variant MeCoₒₚₜMeCoopt requires only one random data for a single forward pass. Based on the proposed zero-cost proxy, we further craft a new zero-shot NAS scheme called FLASHFLASH, which harnesses a new proxy-based operation scoring function and a greedy heuristic. Compared to the existing methods, FLASHFLASH is highly efficient and can construct diverse model architectures instead of repeated cells. We design comprehensive experiments and extensively evaluate our designs on multiple benchmarks and datasets. The experimental results show that our method is one to six orders of magnitudes more efficient than the state-of-the-art baselines with the highest model accuracy.