SRAG: Speech Retrieval Augmented Generation for Spoken Language Understanding

Retrieval augmented generation (RAG) has shown promise for enhancing natural language understanding (NLU) capabilities of large language models (LLMs) by retrieving relevant knowledge as prompts. Extending RAG to spoken language understanding (SLU) represents an important research direction. This paper proposes a RAG approach for improving SLU. First, the encoder of a pretrained automatic speech recognition model is utilized for speech retrieval over the training set. The corresponding texts and intent labels are then formulated as prompts to guide the SLU decoder. Furthermore, a prompt attention mechanism is introduced to strengthen the attention between generation and prompts. Experiments demonstrate that the proposed speech RAG approach substantially outperforms conventional end-to-end and cascaded SLU models in intent prediction from speech. This highlights the efficacy of leveraging retrieval-based prompting to incorporate external knowledge for advancing SLU.