Deep Learning for Audio–Visual Segmentation: A review of models, datasets, and challenges

Audio-Visual Segmentation (AVS) produces pixel-level masks for sound-producing objects in video, bridging the audio and visual modalities. This article reviews the AVS field, covering its core challenges which include mixed audio, visual dominance, cross-modal alignment, multi-source disentanglement, dataset bias, audio hallucination, and off-screen sound. We formally define its four sub-tasks: Audio-Visual Object Segmentation (AVOS), Semantic Segmentation (AVSS), Instance Segmentation (AVIS), and Referring AVS (Ref-AVS). We categorise modern methods into four architectural paradigms (Encoder-Fusion-Decoder, Encoder-Query-Decoder, Encoder-Latent-Decoder, and Encoder-Prompt-Decoder) and compare them from a representation-learning perspective that clarifies how each routes information between modalities. We further synthesise emerging directions that cut across this taxonomy, including contrastive and complementary learning, efficient cross-modal adaptation, text-bridged alignment, reasoning-enhanced segmentation, weakly supervised and training-free AVS, and robustness with reference-free mask quality assessment. A comparative analysis over benchmark datasets and metrics shows that Transformer-based encoders consistently outperform CNNs, the Encoder-Prompt-Decoder paradigm leads on multi-source binary segmentation while fusion paradigms with semantic supervision retain the lead on AVSS, and Multimodal Large Language Models are increasingly central as reasoning engines for Ref-AVS. Through this combination of formal definitions, paradigm-level analysis, challenge synthesis, and empirical comparison, our survey aims to serve as a useful reference for researchers entering or actively working in the AVS field.