Interaction-aware Shared Scene Synthesis for VR Telepresence

Virtual reality telepresence requires immersive shared virtual environments for real-time remote collaboration across different physical scenes. It supports a wide range of applications in teleconferencing, education, and interactive simulations. However, challenges persist in identifying optimal shared virtual spaces that accommodate diverse user interaction requirements while adhering to local physical constraints during scene synthesis. In this paper, we propose an interaction-aware shared virtual scene synthesis method, which uses the large language model (LLM) to produce collaborative virtual scenes based on interaction demands from remote users in different local spaces. First, we introduce the concept of Interaction Aware Template (IAT) and its construction method using an LLM planner. Then, we propose an IAT-based affordance field alignment method for merging the local spaces of the remote users, maximally ensuring that the aligned space could support the user's desired interaction. Finally, we propose an LLM-based shared scene synthesis method according to the merged affordance field. Experiment results show that, compared to existing text-based scene synthesis and mutual space matching methods, our method achieves better Affordance Consistency, 3D Intersection over Union, and Layout Suitability on both scanned and synthesized datasets. The results of the user study demonstrate that the user's subjective perception of interaction fitness and sense of safety were significantly improved.