Weed recognition and localization based on RGB-D object detection framework for weeding robot in Paeonia lactiflora Pall. fields

The unchecked and rapid proliferation of weeds poses a substantial threat to Paeonia lactiflora Pall. (PLP) cultivation. Weed management in PLP fields currently relies primarily on manual weeding, which is labor-intensive, costly and inefficient. Intelligent weeding is a key enabler for advancing the Chinese medicinal herb industry toward sustainable production. However, field environments exhibit numerous uncontrolled factors, such as complex backgrounds, occlusion and overlap among weeds, and lighting variability. It is difficult for single-modality RGB vision systems to achieve precise and robust weed detection. To address these challenges, a depth-guided, single-stage RGB-D weed detection Transformer, termed FW-DETR, was proposed. The framework aims to efficiently utilize spatial geometric priors to guide and fuse visible features. FW-DETR adheres to the modality-graded feature processing paradigm, and designs a Cross-Modal Hierarchical Feature Interaction module (CM-HFI) to form the RGB-D feature encoder, which implements multi-scale bimodal feature interaction and fusion in complex scenes. The proposed Cross-Modal Global Context Modeling module (CM-GCM) enhances spatial awareness and global context capabilities, whereas the Spatially Adaptive Local Feature Modeling module (SA-LFM) recalibrates RGB features and refines local details. The developed Cross-Modal Cascaded Attention module (CM-CAM) incorporates spatial information into high-level features, improving the representation of critical region features. Subsequently, FW-DETR was integrated into the vision system of weeding robot to generate robot-executable 3D spatial operation points. Experimental results show that FW-DETR exhibits good robustness and generalization across diverse field environments, achieving 84.4% mAP 0.5 and 51.7% mAP 0.5:0.95 . The mean pixel positioning errors ( Δ x , Δ y ) of the weeding operation points are 2.57 and 2.61, respectively. Field trial results indicate that the weed removal rate and PLP damage rate are 89.5% and 2.3%, respectively. These results provide a reliable foundation and practical guidance for the development of intelligent weeding equipment for Chinese medicinal herbs.