YOLOv8n-Seg-Based Grape Berry Instance Segmentation and Thinning Decision-Making for Vineyard Robots

Berry thinning is a fundamental operation in modern vineyard management, and future robotic thinning systems have the potential to reduce labor intensity and improve operational consistency. However, automated berry thinning under field conditions is still constrained by insufficient berry-level segmentation accuracy, difficulty in recognizing occluded berries, and high missed-detection rates for small berries. These limitations mainly arise from dense berry arrangements, severe mutual occlusion, and the subtle visual features of small targets. To address these challenges, this study developed a lightweight grape berry instance segmentation and thinning decision-support method based on YOLOv8n-seg. A two-stage knowledge distillation strategy, using Mask R-CNN and YOLOv8l-seg as teacher models, was combined with 30% backbone pruning to improve the recognition of occluded and small berries while maintaining model efficiency. Subsequently, the DBSCAN clustering algorithm was used to analyze berry centroid coordinates and equivalent diameters extracted from instance segmentation masks, thereby generating preliminary thinning-target recommendations based on local berry density and berry size. The model was trained and evaluated on a self-constructed dataset containing 330 valid grape bunch images collected in 2025 from Yongming Vineyard, Lin’an District, Hangzhou, Zhejiang Province, China. The results showed that the optimized YOLOv8n-seg model achieved a box mAP50-95 of 0.8945 and a mask mAP50-95 of 0.7910, with an inference speed of 119.19 FPS and 3.26 M parameters on an NVIDIA RTX 3060 Laptop GPU. Compared with the original YOLOv8n-seg model, the optimized model improved mask mAP50-95 by 1.20 percentage points, increased inference speed by 71.79 FPS, and reduced the number of parameters by 2.38 M. These results indicate that the proposed method improves grape berry instance segmentation performance while achieving a favorable balance among segmentation accuracy, lightweight characteristics, and inference efficiency. The proposed framework provides an offline RGB-based visual perception and preliminary thinning decision-support method for future grape berry thinning robots. However, because the current dataset was collected from Shine Muscat grape bunches at the berry enlargement stage in a single vineyard using the same imaging setup, the results should be interpreted as preliminary evidence under the specific cultivar, growth stage, vineyard, and imaging conditions of this study. Further validation across different grape cultivars, growth stages, vineyards, production seasons, camera systems, embedded platforms, and real robotic thinning operations is still required.