OmniHands: Robust Motion Capture of Interactive Hands via A Versatile Transformer

In this paper, we introduce OmniHands, a universal approach to recovering interactive hand motions and their relative movement from monocular or multi-view inputs. Our approach addresses three major limitations of previous methods: lacking a unified solution for handling single hand and interactive hands image inputs, lacking a versatile framework to support various 4D tasks such as video-based tasks and multi-view tasks, and unable to maintain robust performance in real-world data. To overcome these challenges, we develop a universal architecture with novel tokenization and contextual feature fusion strategies, capable of adapting to a variety of tasks. Specifically, we propose a Relation-aware Two-Hand Tokenization (RAT) method to embed positional relation information into the hand tokens. In this way, our network can handle both single-hand and two-hand inputs and explicitly leverage relative hand positions, facilitating the reconstruction of intricate hand interactions in real-world scenarios. As such tokenization indicates the relative relationship of two hands, it also supports more effective feature fusion. To this end, we further develop a 4D Interaction Reasoning (FIR) module to fuse hand tokens in 4D with attention and decode them into 3D hand meshes and relative temporal movements. The efficacy of our approach is validated on several benchmark datasets. The results on in-the-wild videos and real-world scenarios demonstrate the superior performances of our approach for interactive hand reconstruction. The code and video results can be found in the supplementary materials.