Blueberry flow and mass estimation from harvester conveyor videos via foundation-model-assisted labeling and vision-based sensing

Despite rising labor costs and growing labor shortages, adoption of machine harvesting in fresh market blueberry production has been limited by issues such as high rates of bruising and the significant loss of fruits on the ground. Machine settings strongly affect fruit quality and harvesting performance, yet they are manually configured based on operator experience, with no real-time performance feedback. Towards addressing these limitations, this paper presents a sensing proof-of-concept for analyzing berry flow through the harvester by estimating the pixel-wise proportion of blue vs green berries and volumetric flow rate from overhead RGB-D images of the harvester’s conveyor. An automated labeling pipeline combining a domain-specific YOLOv8 patch detector with SAM2-based mask generation was developed to generate a realistic large-scale berry dataset from unlabeled images captured during commercial over-the-row blueberry harvesting operations. Compared to a baseline SegFormer model trained on a smaller dataset of manually annotated images, our fine-tuned SegFormer – trained and validated on the large-scale automatically generated dataset – achieves superior performance, with green berry IoU improving from 0.377 to 0.817 and recall increasing from 0.414 to 0.911. A scene-level YOLOv8-l detection model trained and evaluated on the same dataset achieves mAP(50-95) of 0.966 and 0.898 for blue and green berries, respectively. Both models’ performance is further confirmed by an independent evaluation on a manually annotated subset of the validation images. We also compared two approaches for predicting harvest yields across five machine-setting trials: a depth-based method using the RGB-D sensor and a detection-based method using only RGB images, achieving R 2 = 0.989 and R 2 = 0.894 , respectively, against ground-truth load cell measurements from these trials. To support future research, we publicly release our datasets and models.