IntegraPSG: Integrating LLM Guidance with Multimodal Feature Fusion for Single-Stage Panoptic Scene Graph Generation

Panoptic scene graph generation (PSG) aims to simultaneously segment both foreground objects and background regions while predicting object relations for fine-grained scene modeling. Despite significant progress in panoptic scene understanding, current PSG methods face challenging problems: relation prediction often only relies on visual representations and is hindered by imbalanced relation category distributions. Accordingly, we propose IntegraPSG, a single-stage framework that integrates large language model (LLM) guidance with multimodal feature fusion. IntegraPSG introduces a multimodal sparse relation prediction network that efficiently integrates visual, linguistic, and depth cues to identify subject–object pairs most likely to form relations, enhancing the screening of subject–object pairs and filtering dense candidates into sparse, effective pairs. To alleviate the long-tail distribution problem of relations, we design a language-guided multimodal relation decoder where LLM is utilized to generate language descriptions for relation triplets, which are cross-modally attended with vision pair features. This design enables more accurate relation predictions for sparse subject–object pairs and effectively improves discriminative capability for rare relations. Experimental results show that IntegraPSG achieves steady and strong performance on the PSG dataset, especially with the R@100, mR@100, and mean reaching 38.7%, 28.6%, and 30.0%, respectively, indicating strong overall results and supporting the validity of the proposed method.