Integrated vision and tactile-based soft robotic system for automated fruit classification and handling in storage environments

Individual assessment and handling of agricultural products is a delicate task, that requires gathering information about the product’s condition, applying specific selection criteria, and manipulate it without causing damage. This remains challenging for industrial robots, particularly when fast, accurate, and non-invasive interaction is required. To address this, the proposed approach combines visual data from a camera with tactile data collected by an instrumented soft gripper. The tactile time-series data acquired during grasping is used for classification. A comprehensive comparison including 18 state-of-the-art machine learning and deep neural network models was conducted, allowing to identify the most suitable for tactile classification using the time series alone. For visual classification, GoogLeNet, AlexNet and Visual Geometry Group Network (VGGNet, also known as Very Deep Convolutional Network) were evaluated under various training parameters. The system classifies products into five fruit categories, each with a fresh or rotten label, for a total of 10 output classes. Tactile data alone achieved limited classification accuracy, with the Diverse Representation Canonical Interval Forest Classifier (DrCIF) showing the best performance at 65.75%, while visual data processed through GoogLeNet achieved 98.91% accuracy. Combining both modalities through a simple fusion layer further improved accuracy to 99.05%, using GoogLeNet for visual data and a Support Vector Machine (SVM) model for tactile time series data. This demonstrates that fusing these complementary sensor modalities significantly enhances classification performance. The feasibility of this approach was demonstrated in real-time, where the algorithm successfully guided a robotic sorting task for the product categories considered in this study.