What type of study is this?

This is a Quantitative Study study (also classified as: Experimental Study).

October 12, 2025Open Access

Continuous Audio Language Models

Key Points

Continuous audio language models generate higher quality audio at a lower computational cost than traditional models.
Experiments reveal that CALM achieved improved efficiency and fidelity in audio generation tasks, surpassing state-of-the-art methods.
The study utilizes transformer architecture to produce contextual embeddings for enhanced audio frame generation.
Results indicate a significant trade-off between audio quality and computational resources with traditional discrete audio language models.

Abstract

Audio Language Models (ALM) have emerged as the dominant paradigm for speech and music generation by representing audio as sequences of discrete tokens. Yet, unlike text tokens, which are invertible, audio tokens are extracted from lossy codecs with a limited bitrate. As a consequence, increasing audio quality requires generating more tokens, which imposes a trade-off between fidelity and computational cost. We address this issue by studying Continuous Audio Language Models (CALM). These models instantiate a large Transformer backbone that produces a contextual embedding at every timestep. This sequential information then conditions an MLP that generates the next continuous frame of an audio VAE through consistency modeling. By avoiding lossy compression, CALM achieves higher quality at lower computational cost than their discrete counterpart. Experiments on speech and music demonstrate improved efficiency and fidelity over state-of-the-art discrete audio language models, facilitating lightweight, high-quality audio generation. Samples are available at hf.co/spaces/kyutai/calm-samples

Read Full Paperexternally

Perguntar à IA

Bookmark

View Full Paper