Programming contact tasks using a reality-based virtual environment integrated with vision

We present an integrated system in which an operator uses a simulated environment to program part-mating and contact tasks. Generation of models within this virtual environment is facilitated using a fast, occlusion tolerant, 3D grey-scale vision system which can recognize and accurately locate objects within the work site. A major goal of this work is to make robotic programming easy and intuitive for untrained users working with standard desktop hardware. Simulation offers the ease-of-use benefits of "programming by demonstration", coupled with the ability to create a programmer-friendly virtual environment. Within a simulated environment, it is also straightforward to track and interpret an operator's actions. The simulator models objects as polyhedra and implements full 3D contact dynamics. When a manipulation task is completed, local planning techniques are used to turn the virtual environment's motion sequence history into a set of robot motion commands capable of realizing the prescribed task.