Foundation model-driven framework for human–object interaction prediction with segmentation mask integration

In this work, we introduce Segmentation to Human–Object Interaction (Seg2HOI), a novel framework that integrates segmentation-based vision foundation models with the human–object interaction (HOI) task, distinguished from traditional detection-based HOI methods. Our approach enhances HOI detection by not only predicting the standard triplets but also introducing quadruplets, which extend HOI triplets by including segmentation masks for human–object pairs. More specifically, Seg2HOI inherits the properties of the vision foundation model (e.g., promptable and interactive mechanisms) and incorporates an implicit relation-learning decoder that applies these attributes to the HOI task. By freezing the vision foundation model, Seg2HOI preserves its original capabilities while using pixel-level mask guidance to support high-level relational reasoning. Despite being trained only for the HOI task, without additional training mechanisms for these properties, the framework demonstrates that such features still operate efficiently. Extensive experiments on two public benchmark datasets demonstrate that Seg2HOI achieves performance comparable to state-of-the-art methods, even in zero-shot scenarios. Lastly, we propose that Seg2HOI can generate HOI quadruplets and interactive HOI segmentation from novel text and visual prompts that were not used during training, making it versatile for a wide range of applications by leveraging this flexibility.