What type of study is this?

This is a Quantitative Study study.

September 29, 2025Open Access

LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

PMPooneh MousaviConcordia University SGShubham GuptaSawai ManSingh Medical College and Hospital CSCem SubakanMila - Quebec Artificial Intelligence Institute

Key Points

LiSTEN enhances performance in audio-language tasks while maintaining fewer trainable parameters, improving efficiency and effectiveness.
The framework utilizes a dynamic prompt selection strategy that balances general knowledge with task-specific information, reducing overfitting risks.
Analysis of selected prompts enhances interpretability, providing insights into how models adapt across varying audio tasks.
By minimizing the need for large-scale datasets, LiSTEN simplifies the training process for audio applications and reduces resource dependency.

Abstract

Foundation models based on large language models (LLMs) have shown great success in handling various tasks and modalities. However, adapting these models for general-purpose audio-language tasks is challenging due to differences in acoustic environments and task variations. In this work, we introduce LiSTEN Learning Soft Token Embeddings for Neural Audio LLMs), a framework for adapting LLMs to speech and audio tasks. LiSTEN uses a dynamic prompt selection strategy with learnable key-value pairs, allowing the model to balance general and task-specific knowledge while avoiding overfitting in a multitask setting. Our approach reduces dependence on large-scale ASR or captioning datasets, achieves competitive performance with fewer trainable parameters, and simplifies training by using a single-stage process. Additionally, LiSTEN enhances interpretability by analyzing the diversity and overlap of selected prompts across different tasks.

KI fragen

Bookmark

View Full Paper