Dual-variant multi-shift hybrid attention for affordance segmentation: two-stage accuracy and one-stage efficiency

This paper tackles the challenges of affordance segmentation in complex multi-object, multi-function scenarios, particularly addressing issues such as occlusion, boundary ambiguity, and semantic confusion. To overcome these challenges, we propose a comprehensive solution that integrates model architecture design, attention mechanisms, and real-time inference optimization. Specifically, we develop two flexible frameworks based on Multi-Shift Hybrid Attention (MSHA) modules configured in both sequential and parallel forms to enhance segmentation performance. For high-precision requirements, we introduce SMSHA-Segmenter, a two-stage system that employs a sequential MSHA-enhanced decoder alongside RTMDet as the object detector. For real-time applications, we present RTMAff, an end-to-end extension of RTMDet that embeds both sequential and parallel MSHA modules to improve semantic consistency and multi-scale adaptability. Experimental results on the IIT-AFF dataset demonstrate the effectiveness of our approach: the two-stage SMSHA-Segmenter achieves an accuracy of 84.62%, while the one-stage RTMAff[X] and RTMAff[tiny] reach approximately 83% and 79% accuracy, respectively, with RTMAff[tiny] operating at 28.7 FPS on an RTX TITAN GPU. Ablation studies further validate the contribution of the MSHA modules. In summary, the proposed architectures achieve a robust balance between accuracy and efficiency, offering a practical and scalable solution for real-time affordance segmentation in complex environments.