RGB-D feature fusion network for rotated rice panicle detection and field yield estimation

Yield estimation is a key procedure in rice breeding. Most traditional prediction methods rely on indirect indicators such as spectral data and biomass. As the most important reproductive organ and phenotypic trait that directly reflects panicle morphology, rice panicles can effectively guide rice yield estimation through the accurate acquisition of their size. However, due to the irregular spatial distribution characteristics of rice panicles, such as random tilting and bending in densely planted environments, the 2D feature information provided solely by RGB images is insufficient for accurate panicle identification and size measurement. Therefore, this study proposes an RGB-D feature fusion rotated rice panicle detection network (RDFFR-Net) and combines it with machine learning models to achieve field yield estimation for new rice varieties. Specifically, RDFFR-Net first uses two feature extractors with identical structures to extract multi-scale features from RGB and depth images. Then, to suppress the interference of depth noise and fully utilize depth features, a cross-attention-based fusion module is designed to adaptively fuse cross-modal features through dynamic weights. Finally, the object prediction module is adopted to further process and extract the multi-scale fused features, thereby achieving accurate prediction of rice panicle positions. On this basis, rotated bounding boxes are utilized to extract the geometric size of rice panicles, which are then used as inputs to construct machine learning models for panicle weight prediction. Experimental results on a self-built RGB-D rotated panicle dataset demonstrate that RDFFR-Net outperforms single-modal and common multimodal detection networks, with mAP@0.5 of 89.5%. In the field yield estimation experiment, the estimation error of a single rice variety using the proposed method was as low as 3.71%. In summary, this study provides a feasible method for field yield estimation of new rice varieties, which has positive theoretical and engineering application significance for promoting high‑throughput phenotyping and crop yield estimation research in the field of smart agriculture.