Non-destructive thermal imaging for object detection via advanced deep learning for robotic inspection and harvesting of chili peppers

Deep Learning has been utilized in computer vision for object detection for almost a decade. Real-time object detection for robotic inspection and harvesting has gained interest during this time as a possible technique for high-quality machine assistance during agriculture applications.We utilize RGB and thermal images of chili peppers in an environment of various amounts of debris, pepper overlapping, and ambient lighting, train this dataset, and compare object detection methods. Results are presented from the real-time and less than real-time object detection models. Two advanced deep learning algorithms, Mask-Regional Convolutional Neural Networks (Mask-RCNN) and You Only Look Once version 3 (YOLOv3)are compared in terms of object detection accuracy and computational costs. When utilizing the YOLOv3 architecture, an overall training mean average precision (mAP) value of 1.0 is achieved. Most testing images from this model score within a range from 97 to 100% confidence levels in natural environment. It is shown that the YOLOv3 algorithm has superior capabilities to the Mask-RCNN with over 10 times the computational speed on the chili dataset. However, some of the RGB test images resulted in low classification scores when heavy debris is present in the image. A significant improvement in the real-time classification scores was observed when the thermal images were used, especially with heavy debris present. We found and report improved prediction scores with a thermal imagery dataset where YOLOv3 struggled on the RGB images. It was shown that mapping temperature differences between the pepper and plant/debris can provide significant features for object detection in real-time and can help improve accuracy of predictions with heavy debris, variant ambient lighting, and overlapping of peppers. In addition, successful thermal imaging for real-time robotic harvesting could allow the harvesting period to become more efficient and open up harvesting opportunity in low light situations.