Symmetry-based 3D shape completion for fruit localisation for harvesting robots

Fruit localisation is a crucial step in developing a robotic fruit-harvesting system. This paper aims to improve the localisation accuracy of fruits in 3D space. In the machine vision system of a harvesting robot, in a single view the visible area of a target is often incomplete and therefore, cannot be directly used to accurately determine the target location. A 3D shape completion method is proposed that can be used on the partially visible images of strawberries obtained from a single view. This method proposed a given number of symmetric plane candidates based on the assumption that the targets are symmetrical, which is normally true for fruits such as such apples, citrus fruits and strawberries. Corresponding rating rules were proposed to select the optimal symmetry to be used for the shape completion. The algorithm was then tested on reconstructed point clouds and implemented on a strawberry harvester equipped with a Red Green Blue-Depth (RGB-D) camera. The evaluation on reconstructed strawberry data showed that the intersection over union (IoU) and centre deviation between the results obtained by this method and ground truth were 0.77 and 6.9 mm, respectively, whilst those of the unprocessed partial data were 0.56 and 14.1 mm. The evaluation results of the strawberry data captured with the RGB-D camera showed that the IoU and centre deviation between the results obtained by this method and ground truth were 0.61 and 5.7 mm, respectively, whilst those of the unprocessed partial data were 0.47 and 8.9 mm.