High-Throughput CNN Inference on Embedded ARM Big.LITTLE Multicore Processors

IoT Edge intelligence requires Convolutional Neural Network (CNN) inference take place in the edge devices itself. ARM big.LITTLE architecture is at the of prevalent commercial edge devices. It comprises of single-ISA cores grouped into multiple homogeneous clusters that enable and performance trade-offs. All cores are expected to be simultaneously in inference to attain maximal throughput. However, high communication involved in parallelization of computations from convolution kernels clusters is detrimental to throughput. We present an alternative called Pipe-it that employs pipelined design to split convolutional across clusters while limiting parallelization of their respective to the assigned cluster. We develop a performance-prediction model that only the convolutional layer descriptors to predict the execution time each layer individually on all permitted core configurations (type and). Pipe-it then exploits the predictions to create a balanced pipeline an efficient design space exploration algorithm. Pipe-it on average in a 39% higher throughput than the highest antecedent throughput.