Stereo vision of liquid and particle flow for robot pouring

We explore stereo vision for recognizing liquid and particle flow as 3D points (a point cloud). In our pouring research [1], we noticed that we could detect liquid flow using optical flow detection, especially with the Lucas-Kanade method [2]. In this paper we extend this idea so that we can reconstruct 3D liquid flow from a stereo camera in order to learn dynamical models of flow. Such dynamical models would be useful to reason about pouring behaviors. We demonstrate our method in pouring various materials: water, coke, jelly, dish liquid, and creamer powder. The results show that our method could detect the 3D flow as a point cloud, and they captured the actual flow phenomenon. We also show that our method works in a robot pouring scenario. Accompanying video: https://youtu.be/2oFjVJwXhKs.