Answers are grounded in this paper and cite their sources.
Evaluates task knowledge extraction from video for collaborative robots, suggesting effective skill acquisition in manufacturing.
In human-centric manufacturing, collaborative robots (cobots) are increasingly expected to continually acquire knowledge as products and workflows evolve, and learning from human demonstration videos has emerged as a flexible means to this end. Realizing this paradigm requires perception that can be deployed with minimal hardware overhead and that generates executable task knowledge for robotic manipulation. Existing video-based methods rarely satisfy both requirements. Geometric pipelines rely on CAD models or multi-camera rigs, hindering scalable deployment across diverse workpieces, while pre-trained vision-language models require demonstration corpora rarely available in high-mix, low-volume manufacturing. In addition, existing methods recover only low-level geometry or high-level semantic labels in isolation, leaving a gap between geometric perception and executable task knowledge. To address these limitations, we propose Video2Knowledge, a video-based learning framework that takes a single monocular RGB video as input and distills it into temporally consistent task knowledge, covering action sequences, object state transitions, and spatial relations, without requiring additional sensors, manual CAD models, or task-specific retraining. The framework couples a physics-aware 6D object pose corrector with hand–object interaction analysis, and consolidates the results into a structured knowledge representation that maps directly to robot action sequences. Experimental evaluation on the HA-ViD and Assembly101 datasets demonstrates that the physics-aware correction yields improved pose coherence, while the extracted task knowledge faithfully recovers demonstrated action semantics and relative geometric configurations. A real-world case study further validates the transfer of task knowledge from a monocular demonstration video to physical cobot execution, confirming the practical deployability in manufacturing scenarios.
No takes yet. Share an insight, caveat, or question.
Huang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: