Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances.This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding.To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes.Our method is minimal yet impactful, outperforming baseline models that require structural modifications.We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data.We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs.Our experiments reveal that incorporating speech descriptions leads to an improvement of nearly 10 points in the F1 score under the zero-shot setting and over 2.5 points under the LoRA setting on the IEMO-CAP dataset.