Semantic labeling of 3D point clouds with object affordance for robot manipulation

When a robot is deployed it needs to understand the nature of its surroundings. In this paper, we address the problem of semantic labeling 3D point clouds by object affordance (e.g., `pushable', `liftable'). We propose a technique to extract geometric features from point cloud segments and build a classifier to predict associated object affordances. With the classifier, we have developed an algorithm to enhance object segmentation and reduce manipulation uncertainty by iterative clustering, along with minimizing labeling entropy. Our incremental multiple view merging technique shows improved object segmentation. The novel feature of our approach is the semantic labeling that can be directly applied to manipulation planning. In our experiments with 6 affordance labels, an average of 81.8% accuracy of affordance prediction is achieved. We demonstrate refined object segmentation by applying the classifier to data from the PR2 robot using a Microsoft Kinect in an indoor office environment.