Low-Rank Attention Reparameterization for Parameter-Efficient Adaptation of the Segment Anything Model to Colorectal Polyp Segmentation

Colorectal polyp segmentation requires accurate boundary delineation, but adapting large vision foundation models to endoscopic images can be computationally expensive. This study investigates whether the Segment Anything Model (SAM) can be specialized for polyp segmentation by updating only a small attention subspace. We propose PolypSAM-Lite, which freezes the SAM vision backbone and applies low-rank reparameterization only to the fused Query–Key–Value attention projections. The model uses bounding-box prompts, binary cross-entropy plus Dice loss, AdamW optimization, and evaluation on Kvasir-SEG with external testing on CVC-ClinicDB and ETIS-LaribPolypDB. PolypSAM-Lite updates 4.2 million parameters and achieves a Dice Similarity Coefficient of 0.9507 on Kvasir-SEG, compared with 0.8804 for zero-shot SAM. External Dice scores are 0.9271 on CVC-ClinicDB and 0.9198 on ETIS-LaribPolypDB, indicating cross-dataset generalization under domain shift. These results suggest that QKV-restricted low-rank attention reparameterization can provide an efficient and effective strategy for adapting SAM to colorectal polyp segmentation without full-backbone fine-tuning.