Region Mining and Refined Query Improved HOI Detection in Transformer

Compared with the object detection task, the Human-Object Interaction (HOI) detection is more complicated. There are interactions between humans and objects, but multiple pairs of interactions may overlap, causing the model to mine features in inappropriate regions and make incorrect predictions. Meanwhile, HOI detection is a multi-task learning task. It requires queries to provide dynamic and correct guided information, thereby better adapting to different images for detecting humans, objects, and interactions. However, the existing query design is fixed and lacks guiding information. To solve this problem, this paper proposes Region Minning and Refined Query (RMRQ). Firstly, the Ground Truth Mask Denoise (GTMD) adopts a denoise training strategy to simulate the overlapping and occlusion problem so that the model can mine more diverse features in the region, avoid mining wrong features, and have the characteristics of denoise to accelerate the training process. Secondly, the Dynamic Linguistic Query (DLQ) dynamically adapts to the input images based on the text information of the HOI triplet to generate queries with guidance information. Meanwhile, Multi-label Focal Loss (MFL) is proposed to constrain the generation of text information to ensure the quality of the query. Finally, extensive experiments on V-COCO and HICO-DET datasets demonstrate the excellent performance of our network and the effectiveness of the above modules.