Stereo vision of liquid and particle flow for robot pouring

We explore stereo vision for recognizing liquid and particle flow as 3D points (a point cloud). In our pouring research 1, we noticed that we could detect liquid flow using optical flow detection, especially with the Lucas-Kanade method 2. In this paper we extend this idea so that we can reconstruct 3D liquid flow from a stereo camera in order to learn dynamical models of flow. Such dynamical models would be useful to reason about pouring behaviors. We demonstrate our method in pouring various materials: water, coke, jelly, dish liquid, and creamer powder. The results show that our method could detect the 3D flow as a point cloud, and they captured the actual flow phenomenon. We also show that our method works in a robot pouring scenario. Accompanying video: https://youtu.be/2oFjVJwXhKs.