A Design of Inference Engine for Recurrent Neural Networks Using Block-Circulant Weight Matrices Based on a SoC-FPGA Approach

The inference engine design of recurrent neural networks (RNNs) is increasingly important for edge-AI applications. To improve the operation efficiency of an RNN inference engine with large weight matrix dimensions, previous works have presented the block-circulant weight matrix scheme that reduces the order of weight amounts for each <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> -dimensional matrix block from <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(n²)</tex> to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O(n)</tex> . This paper proposes an RNN inference engine design using a SoC-FPGA platform based on a SW-HW co-design scheme. In our design, the HW unit can perform individual gate operation involving the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> -dimensional matrix-vector multiplication ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> is the block size from 8 to 128) for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> hidden neurons ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</tex> is a multiple of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> ), and the whole inference operation can be constructed through the SW. The proposed RNN inference engine was verified to demonstrate the phoneme classification using the TIMIT dataset. Compared to the previous FFT -based works, our approach lends support to more flexible ( <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$m,\ n$</tex> ) combinations for various RNN models.