EgoScaler: Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Vision

Learning to use tools or objects in common scenes, particularly handling them in various ways as instructed, isa key challenge for developing interactive robots. Training models to generate such manipulation trajectories requiresa large and diverse collection of detailed manipulation demonstrations for various objects, which is nearly unfeasibleto gather at scale. In this paper, we propose a framework that leverages large-scale ego- and exo-centric videodatasets—constructed globally with substantial effort—of Ego-Exo4D to extract diverse manipulation trajectories atscale. From these extracted trajectories with the associated textual action description, we develop trajectory generationmodels based on visual and point cloud-based language models. In the recently proposed egocentric vision-basedhigh-quality trajectory dataset HOT3D, we demonstrate that our models can generate physically plausible 6DoF objectmanipulation trajectories from action descriptions. Furthermore, we validate the practical utility of the extractedtrajectories through real-robot experiments. The results show that our approach outperforms baseline methods inmost settings and achieves the highest overall success rate.