Self-Training Based Image–Text Multimodal Unsupervised Domain Adaptation Segmentation Model for Remote Sensing Images

Deep self-training-based unsupervised domain adaptation (UDA) semantic segmentation methods learn from labeled source domain images and unlabeled target domain images, performing more stably than those based on adversarial training. We propose a self-training-based image–text multimodal unsupervised domain adaptation semantic segmentation model (SIT-UDA) for remote sensing images. Unlike UDA methods, which rely solely on images, SIT-UDA enhances generalization performance by integrating category hint information from textual descriptions with image data to segment images. SIT-UDA employs a teacher–student self-training framework consisting of two components: the teacher multimodal segmentation model, which predicts pseudo-labels for target domain images, and the student multimodal segmentation model, which is trained to learn feature representations from both the source and target domains with guidance from the teacher model. To enhance the adaptability of image–text pretrained models in remote sensing domains, SIT-UDA introduces text prompt tuning to optimize the text features in the student model, and two learning strategies are proposed to optimize the model’s training objectives: One is the entropy-guided pixel-level weighting (EGPW) strategy, which adaptively weights the loss obtained by self-training on target domain images, leveraging the pseudo-labels rationally according to the entropy value at the pixel level. The other is the contrastive text constraint (CTC) strategy, which maximizes the similarity of text features for the same category between teacher and student models while minimizing the similarity of text features across different categories, improving text feature discriminability to promote cross-domain image–text alignment. Experiments in various domain adaptation scenarios among three remote sensing datasets (Potsdam, Vaihingen and LoveDA) demonstrate that the SIT-UDA is superior to the comparative domain adaptation semantic segmentation methods in terms of qualitative and quantitative segmentation results.