MixConvSAM: Improving Segment Anything Model for Smart Construction Site Segmentation

Recently, with the advancement in the field of smart construction site segmentation, there has been a growing interest in Segment Anything Model (SAM). As a foundational visual model, SAM is renowned for its excellent performance in natural image processing. However, it has encountered challenges when applied to smart construction site images, such as low contrast, blurry boundaries, complex shapes, and small-sized objects, resulting in decreased performance and limited generalization capabilities. To overcome these challenges, this paper proposes a novel unifying model called Mixed Convolution Segment Anything Model (MixConvSAM), which aims to apply SAM’s excellent segmentation performance and strong generalization capabilities to the field of smart construction site segmentation. MixConvSAM not only seeks better generalization capabilities, but also achieves cost reduction in deployment by shortening the sequence length of the Vision Transformer (ViT) branch.Specifically, we have introduced a MixConvolution model on top of SAM, incorporating convolutional localization and window-based self-attention mechanisms to effectively complement local information to the ViT image encoder of SAM. Finally, the outputs of the two pathways are added together to produce local features, which are then injected into the ViT encoder. Additionally, we have collected a dataset of construction site images comprising three categories: bare soil, railing, and stairs, totaling approximately 6.5k images and 6.5k masks. Extensive comparative experiments have shown that the MixConvSAM model outperforms existing task-specific models and generic baseline models in both task-specific evaluation and generalization assessment.