Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection

Abstract In this work, we explore the role of synthetic data in improving the detection of Hand-Object Interactions from egocentric images. Through extensive experimentation and comparative analysis on VISOR, EgoHOS, and ENIGMA-51 datasets, our findings demonstrate the potential of synthetic data to significantly improve HOI detection, particularly when real labeled data are scarce or unavailable. By using synthetic data and only 10\% 10 % of the real labeled data, we achieve improvements in Overall AP over models trained exclusively on real data, with gains of +5. 67\% + 5. 67 % on VISOR, +8. 24\% + 8. 24 % on EgoHOS, and +11. 69\% + 11. 69 % on ENIGMA-51. Furthermore, we systematically study how aligning synthetic data to specific real-world benchmarks with respect to objects, grasps, and environments, showing that the effectiveness of synthetic data consistently improves with better synthetic-real alignment. As a result of this work, we release a new data generation pipeline and the new HOI-Synth benchmark, which augments existing datasets with synthetic images of hand-object interaction. These data are automatically annotated with hand-object contact states, bounding boxes, and pixel-wise segmentation masks. All data, code, and tools for synthetic data generation are available at: https: //fpv-iplab. github. io/HOI-Synth/.