Implementation of Reconfigurable CNN-LSTM Accelerator Based on FPGA

Recently, Convolutional Neural Networks-Long Short-Term Memory (CNN-LSTM) hybrid networks have made significant achievements in deep learning applications such as picture descriptions and video detection. In this paper, we design a reconfigurable accelerator, which can support CNN and LSTM. The reconfigurable acceleration unit (PE) includes weight buffer BRAM and DSP Group modules, which are shared in both modes, effectively saving overhead on-chip resources. we use Uniqlo's 2012–2016 daily transaction data as a dataset to train a small-scale CNN-LSTM model. A reconfigurable CNN-LSTM accelerator with the peak throughput of 49.4GOP/s and the energy efficiency ratio of 26.7GOP/W has been implemented on the Virtex-7VC707 FPGA. Compared with CPU, the accelerator achieves about 100 times more speed than the CPU. Compared with GPU, the power consumption of FPGA is only about 10 % of GPU. Therefore, our accelerator performance is better than other software platforms.