A systematic survey of open-ended visual segmentation: taxonomies, challenges and future directions

Visual segmentation is a cornerstone of computer vision, enabling pixel-level scene understanding for critical applications such as autonomous driving and medical imaging. However, traditional visual models operating under the closed-world assumption struggle with dynamic environment changes and emerging unknown objects. To overcome these limitations, research has shifted toward open-ended environments, categorizing into three distinct research directions: Open-Set Segmentation (OSS), Open-Vocabulary Segmentation (OVS), and Open-World Segmentation (OWS). To provide an informative introduction to researchers who are new to this topic, our paper provides a systematic and comprehensive survey on the development of Open-Ended Visual Segmentation. First, a unified reference framework is established for these research categories, while common datasets and evaluation metrics are summarized. Subsequently, existing methodologies are categorized according to their technical hierarchies: OSS focuses on the binary discrimination of unknown classes; OVS leverages large-scale Vision-Language Models (VLMs) for zero-shot generalization via natural language prompts; and OWS emphasizes autonomous discovery and incremental knowledge maintenance. Moreover, key methodological routes are systematically classified. In particular, it covers the uncertainty estimation, vision-language alignment, and knowledge distillation. Meanwhile, this work also provides an in-depth analysis and quantitative performance comparisons of state-of-the-art techniques in these domains. Finally, prevalent challenges including computational efficiency, semantic ambiguity, and the lack of unified benchmarks are discussed, with potential research trajectories outlined to provide an informative survey and a valuable resource to researchers for subsequent technological developments and innovation. The collected methods and datasets will be made publicly available at https://github.com/CVLife/Awesome-Open-ended-Segmentation .