Mamba-enhanced RT-DETR for ship detection in optical remote sensing images

Ship detection from optical remote sensing images remains challenging due to cluttered sea–land backgrounds, large scale variations, low-contrast small targets, and dense ship occlusion. Existing detectors struggle to balance global modeling capability, detection precision, and computational efficiency. This paper proposes a Mamba-enhanced Real-Time Detection Transformer (RT-DETR) framework for high-precision ship detection and identification. First, a Mamba-based Deformable Spatial Attention (Mamba-DSA) backbone is proposed to integrate linear-time state-space feature modeling and deformable spatial attention, which strengthens ship-specific feature representation and suppresses background interference. Second, a Context-Guided Spatial Feature Reconstruction Feature Pyramid Network (CGRFPN) is designed to enhance multi-scale context awareness and rectangular region calibration, effectively reducing missed detections in dense ship scenes. Third, a Spatial-to-Depth Depthwise Convolution (SPD-DWConv) module is developed to enhance small-target features without introducing excessive computation overhead. Experiments on HRSC2016, NWPU VHR-10, and DOTA-v1.5 datasets show that the proposed method outperforms the baseline RT-DETR by 9.0% in precision, 16.5% in recall, 19.0% in mAP@50, and 14.1% in mAP@50:95. The proposed method achieves strong robustness and efficiency, showing high practical value for maritime monitoring, port security, and emergency response.