StrawSnake: A Real-time strawberry instance segmentation network based on the contour learning approach

Abstract Fruit recognition is critical for automated harvesting systems. This paper aims to explore a high-precision, real-time method for strawberry recognition. Existing methods for strawberry recognition are mostly pixel-level and box-based. However, these methods cannot accurately or in real-time determine the position of strawberries. Therefore, this paper proposes a new contour-based real-time detection and segmentation network for strawberries, called "StrawSnake." A strawberry-specific octagonal contour is designed based on the shape of strawberries for initial strawberry contour extraction. Deep snake convolution is used for boundary feature extraction and a contour feature aggregation module is proposed, prioritizing important boundary feature channels and enhancing the global context understanding capability of strawberries by aggregating multi-level strawberry contour features. This study obtained a new high-resolution dataset of 3100 strawberry images and the corresponding manually segmented ground truth images (StrawDI_Db1 database). Edge detection was used to label the ground truth images of strawberry contours in the dataset to build strawberry contour labels. Results obtained from testing 200 images in the dataset show that our StrawSnake can achieve real-time strawberry contour recognition at a speed of 39.2 frames per second, with an average accuracy of 59.23%. On another TongStraw_DB built by us, compared with other methods,, our StrawSnake algorithm has also a competitive advantage in terms of segmentation speed and accuracy. Finally, through ablation experiments, we demonstrate the performance improvement brought by DSConv and MFRB. Since our method can be integrated into strawberry automated harvesting systems, it is considered the most promising approach.