Adapting segment anything model for 3D pancreas tumor segmentation

The Segment Anything Model (SAM) stands as a foundational model for general-purpose image segmentation, showing promising performance in various natural image segmentation tasks. However, its inherent 2D characteristics often lead to inadequacies in capturing and integrating crucial information in 3D space, particularly when dealing with volumetric medical image data. To address this problem, we propose 3D ASAM, a novel approach for volumetric medical image segmentation. Specifically, our model feeds volumetric images directly into the SAM and CNN branches to help the model better understand the complex structure and texture of the images. We introduce a series of innovative 3D adapters in the Transformer block of the image encoder, while retaining most of the pre-trained weights in SAM. We utilize a parametrically efficient fine-tuning strategy and propose a new boundary difference loss function, aiming at improving segmentation accuracy and boundaries for better medical image segmentation. We conducted extensive experiments on pancreas tumor medical image datasets, and our method achieved competitive results compared with other advanced 3D methods when only single-point cues were provided.