Lighted-SAM: Lightening Open-World SAM for Low-light Segmentation

Segment Anything Model (SAM) has achieved impressive segmentation performance in an open-world setting. However, SAM relies heavily on high-quality input images and usually struggles in low-light conditions. This is mainly caused by the pre-training dataset, SA-1B, in which low-light samples constitute a relatively small fraction of the data. This lack of presence leads to a noticeable weakness when SAM is applied in real-world dark environments. With the motivation of improving SAM's performance under low-light conditions while retaining its strong zero-shot capability, this work proposes an alignment stage between the pre-training stage and testing stage. Unlike existing low-light studies that mainly focus on task-specific and close-set settings, our work further emphasizes pursuing the segmentation ability under low-light conditions for open-world models. To this end, we construct DarkSeg58K, a realistic and diverse dataset, which serves as the alignment dataset to support this stage. We further introduce Lighted-SAM as the lightweight repair strategy to fix SAM's performance in low-light conditions. Different from existing methods focusing on introducing spectral adapters into the model design and training this model end-to-end, Lighted-SAM introduces the Spectral Information Resonance (SIR) mechanism to harmoniously integrate the spectral enhancement module into SAM, which is usually kept frozen due to its large-scale parameters. Based on our lightweight repairing strategy, Lighted-SAM can improve SAM's ability in low-light conditions while preserving its zero-shot ability. Experiments on different benchmarks validate the superiority of our approach. Code will be released when this work is fully accepted.