Geoinformatics-oriented concept-guided segmentation of Earth observation imagery using text-driven foundation models

Earth observation image segmentation is a central geoinformatics task, enabling precise land cover mapping, infrastructure monitoring, and environmental change detection from satellite and aerial imagery. However, existing remote sensing segmentation models are typically trained for fixed class taxonomies, lack cross-sensor generalizability, and require extensive manual annotation for every new geographic scene or sensor configuration. To overcome these limitations, we propose ConceptSeg, a concept-driven open-vocabulary segmentation framework tailored for geoinformatics-oriented Earth observation analysis, which we evaluate on optical satellite (Sentinel-2, Gaofen-2, and WorldView-3), aerial, and UAV imagery. ConceptSeg fine-tunes a decoder built on top of a frozen Segment Anything Model 3 (SAM 3)-style Promptable Concept Segmentation (PCS) backbone on a large curated corpus of remote sensing images paired with rich geospatial concept labels, comprising 285,100 image patches drawn from seven publicly available benchmarks. Users can target arbitrary land cover types, buildings, agricultural features, and disaster damage via open-vocabulary text descriptions such as “regulated river channel,” “white residential house,” or “terraced farmland,” without being restricted to predefined class sets or geometric prompts. The framework addresses fundamental domain gaps between natural and overhead imagery—differences in viewing geometry (nadir vs. oblique), spatial resolution (0.3–10 m GSD), spectral rendering, and geospatial terminology—through three purpose-built components: a multiscale image encoder with geospatially aware positional encoding, a concept encoder that projects textual descriptions into geospatial semantic embeddings, and a cross-attention decoder that fuses prompt features with multiscale image representations. Because prompted segmentation methods can receive different amounts of test-time information, we evaluate ConceptSeg under three explicitly separated input regimes: (i) conventional closed-set automatic segmentation, (ii) fully automatic open-vocabulary text-only segmentation, and (iii) interactive segmentation guided jointly by text and a spatial bounding box. Comprehensive experiments on seven remote sensing benchmarks demonstrate that, within each regime, ConceptSeg is competitive with or outperforms specialist CNN/transformer models and SAM-based adaptations evaluated under the same input regime. Under the interactive text-plus-box regime, ConceptSeg T + I achieves 0.929 mIoU on building extraction (WHU Building), 0.579 mIoU on multiclass land cover segmentation (LoveDA), 0.896 mIoU on dense urban buildings (Inria), 0.723 mIoU on agricultural anomaly detection (Agriculture-Vision), and 0.771 mIoU on disaster damage assessment (xView2), each 1.7–3.3 percentage points above the strongest baseline evaluated under the same box-conditioned regime, and stable across three independent training runs (paired-bootstrap test, p < 0.05 ; full mean ± s.d. and per-class statistics in Section 4.5). Under the fully automatic, text-only regime, ConceptSeg T attains only 0.21–0.46 mIoU, confirming that text alone does not reproduce the box-conditioned numbers above and that spatial priors remain important for top-tier accuracy; the framework additionally exhibits open-vocabulary generalization on withheld concept phrases (avg. 0.594 mIoU across LoveDA and Gaofen Image Dataset novel phrases, +15.6 points over the strongest training-free baseline) and cross-sensor transfer from Sentinel-2 to Gaofen-2 and aerial RGB composites (0.712 and 0.685 mIoU with oracle boxes, retaining 92%–94% of this performance when boxes are instead produced by an off-the-shelf detector, Section 4.6). These findings indicate that grounding remote sensing segmentation in semantic language embeddings, combined with spatial priors, improves domain generalization relative to pixel-space baselines, whereas fully automatic text-only segmentation remains comparatively weaker and is an important direction for future work. Code, configuration files, data splits, and trained checkpoints will be released at https://github.com/ConceptSeg/ConceptSeg upon publication.