Infrared Moving Maritime Vessel Segmentation Based on Multi-Scale Spatial–Temporal Transformer Network

Infrared moving maritime vessel segmentation is a crucial image processing task for maritime security, which is a challenging problem due to the complex backgrounds and targets with varying sizes. To address these issues, we propose an end-to-end segmentation network based on a multi-scale spatiotemporal vision transformer (ST-VT) for segmenting the moving maritime vessels in the infrared image sequence. Specifically, in the feature extraction module, we introduce a multi-scale feature encoding structure that combines a multi-scale backbone and Feature Pyramid Network technology. Then, the multi-scale deformable encoder structure and a cross-scale fusion module with the pixel decoder are proposed to generate the multi-scale spatiotemporal features. Subsequently, we employ the improved attention blocks that are the core blocks of the coarse-to-fine framework (across scales) of the prompt decoder to obtain the prompts. Finally, a multi-scale mask decoder is applied to achieve the final target segmentation. The experiments are conducted on the benchmark dataset IPATCH and our labeled dataset LAS-MassMIND. The results demonstrate that the proposed method achieves state-of-the-art performance, especially within complex backgrounds and targets of varying sizes.