ROI-constrained query selection for small object detection with RT-DETRv2

Object detection is a significant research direction within the field of computer vision. In recent years, Transformer-based object detection methods have achieved a commendable balance between detection accuracy and inference efficiency, with RT-DETRv2 demonstrating robust real-time detection capabilities. However, during the decoder query initialization stage, RT-DETRv2 primarily employs a global Top-K selection strategy based on encoder token classification scores. This approach tends to overlook local key regions, particularly in complex backgrounds and scenarios involving small objects, which complicates the effective retention of weak-response targets. To address this limitation, this paper proposes an enhanced method termed ROI-DETR, which is founded on ROI-constrained query selection. This method integrates the ROI Mask pass-through mechanism within the RT-DETRv2 framework and combines it with an adaptive token mask mechanism, a multi-scale budget allocation strategy, and an ROI-guided Mixed Top-K query selection strategy. These enhancements optimize the query generation process from the encoder to the decoder, thereby bolstering the model’s ability to concentrate on potential target regions. Experimental results indicate that on the COCO dataset, the Average Precision (AP) of ROI-DETR reaches 0.501, reflecting an improvement of 0.006 over RT-DETRv2, while the AP for small objects reaches 0.321, an enhancement of 0.015. Furthermore, three repeated experiments on DOTA and TinyPerson provide evidence of cross-dataset applicability. On DOTA, ROI-DETR achieves an AP of 0.259 ± 0.001 compared with 0.251 ± 0.001 for RT-DETRv2; on TinyPerson, the mean AP increases from 0.193 ± 0.002 to 0.197 ± 0.001, while the APs change remains marginal. In the ablation experiments, this paper substantiates the contribution of each sub-module to detection performance and analyzes the operational mechanisms of each module. The statistics and visualization results concerning Top-K token and ground truth (GT) matching further demonstrate that this method can enhance the coverage of target-related queries while mitigating background token interference. In summary, ROI-DETR significantly improves the selection quality of small target-related queries in complex backgrounds, offering a viable approach to optimizing the small target detection performance of real-time Transformer detectors.