QS-NAS: Optimally Quantized Scaled Architecture Search to Enable Efficient On-Device Micro-AI

Because of their simple hardware requirements, low bitwidth neural networks (NN) have gained significant attention over the recent years, and have been extensively employed in the state-of-the-art devices that seek efficiency and performance. Research has shown that scaled-up low bitwidth NNs can have accuracy levels on par with their full-precision counterparts. As a result, there is a trade-off between quantization (q) and scaling (s) of NNs to maintain the accuracy. To capture that trade-off, in this paper, we propose QS-NAS which is a systematic approach to explore the best quantization and scaling factors for a NN architecture that satisfies a targeted accuracy level and results in the least energy consumption per inference when deployed to a hardware–FPGA in this work. We first approximate the accuracy of a NN using a polynomial regression based on experiencing over a span ofqands. Then, we design a hardware that is scalable withPprocessing engines (PE) andMmultipliers per PE, and infer that the configuration of the most energy-efficient hardware as well as its energy per inference for a NNq,\,sare, in turn, a function ofqands. Experiencing the NNs with variousqandsover our hardware, we approximate the energy consumption using another polynomial regression. Given the two approximators, we obtain a pair ofqandsthat minimizes the energy for a given targeted accuracy. The method was evaluated on SVHN, CIFAR-10, and ImageNet datasets trained on VGG-like and MobileNet-192 architectures, and the optimized models were deployed to Xilinx FPGAs for fully on-chip processing. The implementation results outperform the related work in terms of energy-efficiency and/or power consumption, yet having similar or higher accuracy. The proposed optimization method is fast, simple, and scalable to emerging technologies. Moreover, it can be used on top of other AutoML frameworks to maximize efficiency of running artificial intelligence on edge devices.