Single-Domain Generalized Predictor for Neural Architecture Search System

Performance predictors are used to reduce architecture evaluation costs in neural architecture search, which however suffers from a large amount of budget consumption in annotating substantial architectures trained from scratch. Hence, how to leverage existing annotated architectures to train a generalized predictor to find the optimal architecture on unseen target search spaces becomes a new research topic. To solve this issue, we propose a Single-Domain Generalized Predictor (SDGP), which aims to make the predictor only trained on a single source search space but perform well on target search spaces. In meta-learning, we firstly adopt feature extractor in learning the domain-invariant features of the architectures. Then, a neural predictor is trained to map the architectures to the accuracy of the candidate architectures over the target domain simulated on the source search space. Moreover, a novel multi-head attention driven regularizer is designed to regulate the predictor to further improve the generalization ability of the predictor for the feature extractor. A series of experimental results have shown that the proposed predictor outperforms the state-of-the-art predictors in generalization and achieves significant performance gains in finding the optimal architectures with test error 2.40% on CIFAR-10 and 23.20% on ImageNet1k within 0.01 GPU days.