U-Sparta: A Unified Speech Processing Accelerator for Real-Time Applications

With the advancements in deep learning for speech processing and the rising popularity of smart devices, the demand for edge devices to support multiple speech processing applications is steadily increasing. However, existing unified AI platforms are unsuitable for such edge scenarios, while dedicated accelerators lack flexibility for diverse speech tasks, and current multi-task solutions remain limited in operator support and task coverage. In this work, we propose U-Sparta, a Unified Speech Processing Accelerator for Real-Time Applications. First, a decomposition scheme, comprising three dedicated strategies, is proposed to efficiently support various neural network operators for speech processing. Additionally, an efficient hardware-implemented approach for precise layer-wise fixed-point quantization is introduced, and a range of compression techniques are employed; then we propose hardware-aware expansion to effectively compensate for performance degradation caused by compression. These approaches collectively reduce the model size by an average of 93.1% and computational complexity by 89.9%. The design has been implemented on a Pango PGL50H FPGA, achieving worst case 1.65W and 1.39W on average low power consumption at 50MHz, lower resource utilization compared to single-task related works. It has been further implemented in 55-nm CMOS technology, with a core area of 3.61 mm2, achieving an average power consumption of 22.6 mW and high accuracy across multiple speech processing applications, including voice activity detection (VAD), keyword spotting (KWS), speaker verification (SV), environmental sound classification (ESC), speech enhancement (SE), speech separation (SS), and speech recognition (SR).