SAM-Based Input Augmentations and Ensemble Strategies for Image Segmentation

Despite the remarkable progress of deep learning in image segmentation, models often struggle with generalization across diverse datasets. This study explores novel input augmentation techniques and ensemble strategies to improve image segmentation performance. We investigate how the Segment Anything Model (SAM) can produce relevant information for model training. We believe that SAM offers a promising source of prior information that can be exploited to improve robustness and accuracy. Building on this, we propose input augmentation techniques that integrate SAM information directly into the images, enhancing the learning process of segmentation models. Each proposed augmentation method comes with its unique advantages; therefore, to leverage the strengths of each approach, we introduce AuxMix, a model trained with a combination of SAM-based augmentation methods. We conduct experiments on different state-of-the-art segmentation models, evaluating the effects of each method independently and within an ensemble framework. The results show that our ensemble strategy, combining complementary information from each augmentation, leads to robust and improved segmentation performance in a large set of datasets. We use only publicly available datasets in our experiments, and all the code developed to reproduce our results is available online on GitHub.