Abstract Instance segmentation in remote sensing images plays a significant role in fields such as 3D map construction, urban planning, and disaster monitoring. However, the challenges posed by large variations in target scale, blurred edges, and high density in such images make the segmentation task particularly difficult. To address these issues, we propose the MDS-YOLOSeg algorithm for remote sensing image instance segmentation, based on an improved YOLOv11s-Seg model. To tackle problems caused by large target scale variations and blurred edges, which result in insufficient edge perception, we introduce the Multi-Scale Edge Information Enhancement and Selection module (MSEIES), which extracts features from different scales, emphasizes edge details and enhances edge perception capabilities. We also designed a Dual-Edge Information Fusion Module (DEIEM) to preserve important spatial details and reduce background interference. To address the issue of dense targets, we introduce a novel loss module, Soft-NMS-ShapeIoU, which suppresses adjacent interference boxes while maintaining the spatial correspondence between the mask and the bounding box. The experimental results show that the metrics such as AP and AP50 for MDS-YOLOSeg algorithm on the iSAID dataset are 42.7% and 72.6% respectively, which are 4.3% and 6% higher than those of the baseline algorithm. Additionally, under the same conditions, MDS-YOLOSeg achieves an AP of 70.1% on the NWPU VHR-10 dataset, significantly outperforming other methods, indicating that our approach has strong generalization ability and transferability.
Li et al. (2025) studied this question.