A SAM-based dual-branch network for remote sensing semantic segmentation

Semantic segmentation of satellite images plays a crucial role in various remote sensing applications. Recently, the introduction of large models, such as the segment anything model (SAM), has provided a promising approach for this task. However, SAM’s outputs lack specific class information and precise boundary accuracy for remote sensing applications, as it has primarily been trained on natural images without semantic guidance. To leverage SAM’s strengths while addressing its limitations, we propose the SAM-based Dual-Branch Network (SAM-DBnet). This architecture features a dual-encoder system, consisting of a main encoder and a SAM-based auxiliary, to enhance the general feature extraction. The SAM-based auxiliary, known for its robust capability to extract features, is fine-tuned on satellite images, serving as an additional encoder to improve overall feature extraction. The features obtained from both the SAM-based auxiliary and the main encoder are integrated through a fusion module and then processed by a decoder to produce the final segmentation results. Experimental evaluations on two prominent datasets, International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam, confirm the effectiveness and broad applicability of the proposed method.