Machine vision for fruit detection and localization in intelligent harvesting robots: A comprehensive review

Autonomous fruit harvesting requires not only accurate fruit detection but also the reliable transformation of uncertain visual observations into safe, efficient, and low-damage robotic harvesting actions. Achieving this objective remains challenging in commercial orchards because variable illumination, severe fruit occlusion, plant motion, and constrained workspaces introduce substantial uncertainty into both visual perception and robotic execution. In this review, we systematically synthesize recent advances in machine vision for fruit detection, segmentation, three-dimensional localization, pose estimation, and picking-point localization, and examine their integration with end-effector operation and hand–eye coordination in intelligent harvesting robots. We compare eye-in-hand, eye-to-hand, and hybrid vision configurations; two-dimensional and three-dimensional sensing modalities; machine-learning-, deep-learning-, and Transformer-based perception methods; and representative harvesting-system configurations. This review establishes an integrated perspective on the interactions among sensing, perception, motion planning, grasping, detachment, and feedback within a perception–action–interaction cycle. It highlights the transition from object-oriented detection to operation-oriented perception, which involves not only understanding the fruit position but also the attachment structure, approach direction, obstacles, interaction risk, and harvestability. There are several challenges to deployment in complex field conditions, including calibration drift, incomplete spatial information, limited end-effector adaptability, and accumulated perception-execution errors. Future harvesting systems will need to integrate multimodal sensing, active view planning, uncertainty-aware decision-making, visual servoing, and tactile-force feedback to enable autonomous fruit harvesting to be scalable and damage-aware.