Revisiting the Inference Time Optimization of TinyML for ARM-Based Microcontrollers

As AI research advances, various performance optimization techniques have been studied for TinyML models on microcontrollers with very limited resources. In TinyML frameworks such as TFLM (TensorFlow Lite Micro) and NNOM (Neural Network on Microcontrollers), their execution engines mainly depend on CMSIS-NN for inference acceleration, which is an ARM’s back-end library to execute optimized kernel functions for performance optimization. But we often find that CMSIS-NN is not invincible for inference time optimization on TinyML. In this paper, we examine TinyML frameworks and their CMSIS-NN libraries and consider how to improve CMSIS-NN in terms of runtime. Then, we propose the D2I technique to reduce the overhead of memory operations that occur while performing the Im2col procedure within the convolution function, which takes most of the inference time in CMSIS-NN. The proposed technique creates a necessary index table, finds the location of the input with the corresponding index, and performs direct operations between filters and inputs. Thus, it can quite mitigate data copy operations in Im2col with a small additional amount of memory compared to Im2col. In extensive experiments using an Arduino nano 33 BLE board with Cortex-M4 and an STM32F746G-DISCO board with Cortex-M7, D2I was found to achieve about 16.3% and 14.5% inference time improvements against the Im2col in TFLM’s and NNOM’s CMSIS-NNs, respectively, for the SqueezeNet model. And the additional memory usage was shown to be identically 11.52 kB.