Vision-language model guided three-dimensional segmentation for automated and high-fidelity strawberry point cloud reconstruction

For strawberry picking or inspection robots, acquiring three-dimensional (3D) point clouds of strawberries from single-view red, green, and blue (RGB) images or point cloud data is crucial. Current methods rely on a large-scale three-dimensional point cloud library of strawberries. To obtain this library, existing methods mainly rely on Light Detection and Ranging (LiDAR), point cloud registration, and segmentation, which are costly and time-consuming. This research proposes a multi-view driven 3D segmentation framework based on three-dimensional Gaussian Splatting (3DGS) and a vision-language model (VLM) to enable efficient and accurate reconstruction of complete strawberry point clouds. Firstly, an image acquisition device and software are developed, enabling rapid capture of images from multiple viewpoints. Secondly, the Segment Anything Model (SAM) is fine-tuned using Low-Rank Adaptation and its bounding-box prompts are provided by the Vision Transformer for Open-World Localization (OWL-ViT) with “strawberry” concept as the prompt for OWL-ViT; thus, accurate strawberry segmentation masks on raw images are obtained as supervision masks. Finally, a joint loss function is proposed that integrates RGB color consistency loss and edge-aware loss to optimize the 3D segmentation weight vector iteratively. On a self-constructed dataset containing 400 strawberries, each captured from 144 RGB viewpoints, experiments show that the proposed method achieves a mean Intersection over Union (mIoU) of 89.44% and a mean Accuracy (mAcc) of 99.40% in the 3D segmentation task, while maintaining high computational efficiency. Additionally, the superiority of the proposed method is further validated through generalization experiments on several complex scenes (e.g., bonsai, kitchen, garden) from a publicly available dataset.